Encryption
In today's world of research, researchers regularly handle data, send it over the internet and store it in the cloud. At any point, especially when the internet is involved, the data is exposed to some risk. Keeping data safe and encrypted is hence a key component of IRB requirements and research ethics. Encryption should take place whenever dealing with sensitive data in any stage of research: from sampling and data collection to cleaning and analysis. This page discusses encryption in transit and at rest; key pairs; password management; and encryption with SurveyCTO data.
Read First
- There are three main types of encryption algorithms relevant to our team. They are symmetric encryption, asymmetric encryption and hashing.
- When transferring or sharing sensitive data, the data both need to be encrypted-in-transit and encrytped-at-rest
- Encryption keys that can decrypt sensitive data must be shared using password managers
Three types of Encryption
There are three types of encryption useful to understand to safely operate in the digital world and to safely handle PII data. The difference between these three encryption types are under which circumstances the information can be decrypted and what key or password is needed to do so.
The three types are:
- Symmetric Encryption - Information can both be encrypted and decrypted and the same password is used when encrypting and decrypting
- Asymmetric Encryption - Information can both be encrypted and decrypted but different passwords are used when encrypting and decrypting
- One-way hashing - No password is used when encrypting, and there is no way decrypt anything that was one-way hashed
There are different algorithms for each of these different types of encryption, but most people do not need to know that there are different types of algorithms within these three types or what the difference between them are. Some of these algorithms are outdated and can easily be cracked, but unless you are someone implementing an encryption service of some sort, you do not need to know the details. Just google the service you want to use, and see if there are recommendations against using it, or ask someone you trust for which service to use. The recommendations given on this page are to services that have been recommended for a long time and is likely to keep being recommended.
Symmetric Encryption
A symmetric encryption algorithm can both encrypt and decrypt files. When it encrypts it takes two inputs, the file to be encrypted and a key, and the output is the encrypted version of the file. When it decrypts it also use two inputs, the encrypted file and the same key, and the output is the file decrypted. “Key” in this context is almost synonymous with password, but passwords are only one type of key. Advanced encryption may use such long passwords as the key that it becomes a file.
Anyone with access to both the encrypted file and the key can decrypt and read the file, so when services encrypt our data it is never secure enough if they have access to the key. In research, when you have an IRB -- which you should always have if you collect information on individuals — no-one but the people listed in the IRB should have access to the key. This is a central aspect when assessing if an encryption is good enough. For example, services like DropBox and Amazon AWS offer automatic encryption on their servers. The encryption algorithms they are using are state-of-the-art and correctly implemented. However, in order to make it automatic, they keep a copy of the key, and as long as DropBox or Amazon are not included in your IRB, this is not good enough encryption. There is nothing wrong with them encrypting our files automatically, as it means that no-one but them can read the files unless they are hacked, but we also need to exclude service providers when we protect sensitive data.
One extremely important aspect of symmetric encryption is that there is absolutely no way to decrypt the file if the key is lost. If you are using any service that claims to be able to recover information decrypted with symmetric encryption without the key, then they are either lying or that service is insecure.
Use case - secure files to be shared - VeraCrypt If we want to share a file with identifying or otherwise sensitive data over insecure services like e-mail or DropBox, we first need to manually encrypt the file. We can use VeraCrypt (an free open source software for symmetric encryption) to create a “secure folder” that we will compare to a safe, and provide you with a key that opens this safe. Just like a physical safe, anyone with access to the safe who also has the key will be able to access its content. You have to specify how much data the secure folder should be able to store, for example 100MB, and then that folder will take that amount of disk space on your computer regardless of being is empty or full, just the same way as you cannot see from the outside of a physical safe if it is full or empty. You cannot store more data than what you set the limit to when you create the secure folder, so if you run out of space, you must create a new secure folder with a bigger capacity and move the files there.
When using VeraCrypt, anyone who has access to both the encrypted file and the key, and has VeraCrypt installed can decrypt the encrypted file and view, modify or add content. VeraCrypt decrypts a file in such a way that is only decrypted as long as it is actively decrypted. This means that as soon as either you chose to stop decrypting it, you close VeraCrypt, or your computer shuts down or lose power, the file is immediately encrypted again. When a secure folder is no longer decrypted it is always re-encrypted with the key that was used when it was initially created.
==== Once the secure folder is created you can share it empty or with content over insecure ==== connections — like e-mail, DropBox, make it public on the web, etc. — as long as you share the key in a secure fashion. If you share the key in an insecure way, for example by putting it in an e-mail or in DropBox, it would be like shipping a physical safe with the key taped to the outside of it. The secure way to share the key is to use a type of software called password managers. Example of password managers are LastPass (closed source but World Bank approved) and BitWarden (open source but not approved for installation on World Bank computers). You use a password manager to safely share keys
This is very similar to when you use VeraCrypt to encrypt files before sharing them. The only difference is that you simply do not share the key with anyone. You still need the key yourself to decrypt the files later, so we strongly recommend you using a password manager even if you will not share the key with anyone.
Is open source software like VeraCrypt less secure as everyone can see how it is implemented?
Open source is often said to be less secure, but that is not true if you use a software that has been around for a long time and has been examined and used by many cyber security experts. VeraCrypt fulfills both those requirements. When it comes to encryption, it is safer to use scrutinized open source software as VeraCrypt since you can be sure there is no “backdoor” implemented where there is a secret master key that can decrypt all files. Non-open source software are often forced by national intelligence agencies to implement such a “backdoor” so that governments gets access to your encrypted data. But it is not possible to add backdoors in open source software without it being visible to everyone. So if a large community of users trusts an open source software, that is your safest option.
Asymmetric Encryption
While symmetric encryption is like a safe where everyone viewing, modifying or adding content have must have the same key, asymmetric encryption is like the post office’s post collection box where anyone can add content but only the post office has the key that can open and view the content. So instead of one key, in asymmetric encryption there is a public/private key pair. So in the post box example, there would be a public key written on the post box for anyone to use when sending a letter, for example “press 1, 2, 3 on the keypad to open the mail chute” and a private key that only the postal workers would use when they empty all the letters. While you do not risk compromising any data you have already collected by publishing the public key to the world, you should not do that as it would allow a malicious person to send you so much fake data to your collection box that it might be difficult to tell what is real data and what is fake data.
While the main benefit of asymmetric encryption is that you do not share the private key with anyone who sends you data, that is also the main limitation as it is only secure as a one-way communication. Another benefit is that it allows for a more automatic set-up as it is ok for servers to handle sharing of the public key. If you set up the private/public key pair and you want to share the private key with anyone so that they can also decrypt the data, then that private key needs to be shared using a password manager as in symmetric encryption.
Just as with symmetrical encryption, there is absolutely no way to decrypt files if the private key is lost. So even if you do not intend to share the private key with anyone, we still strongly recommend that you store it in a password manager for the future. The public key cannot be re-generated either if lost, but that doesn’t tend to be an issue, as you can create a new digital collection box and use the new private/public key pair.
Use case - sending data from the field
Asymmetric encryption is perfect for data collection as the data is only intended to flow one way, and we do not want to have to set up password managers on the devices used in data collection. By using asymmetric encryption we can allow the tablets to send data securely to the server without making it possible for anyone using a tablet to see what is already on the server. Just as in symmetric encryption it is important that no-one who is not on the IRB has access to the private key, so the private key cannot be shared with the data collection service. The private key should not be used to decrypt the data while it is still on the server, it should only be decrypted when or after it is downloaded. If you did not set up encryption or if you decrypted your data while it was still on the server, then the data collection service provider can read your data. So can also any server admins and anyone hacking into the data collection server.
When you download the data you should decrypt it with the private key, and then put it in a folder that is encrypted using symmetric encryption. The reason we shouldn’t keep using the asymmetric public/private key pair is that we need different keys to view and add data and there is no practical way of modifying data already encrypted. In symmetric encryption the same key is used to view, modify and add data, which is much more practical when you start working with your data.
Use case - secure communication online - http vs https
There are two main differences between http and https, and asymmetric encryption is very central to one of them. When you communicate with a server over https, you first send a message with no sensitive data to the server saying that you are ready to send something in a secure way. The server then creates a public/private key pair and sends the public key to you. Your browser then encrypts your data using the public key and sends it to the server. The server then decrypts it with the private key, which is kept secret as it never left the server. That specific public/private key pair is then discarded to never again be used. This is repeated thousands of times each time you use the internet.
The other main difference between http vs https is that the server signs the response with the public key, so that your browser knows that the response truly came from google.com or whatever site you are talking to. Modern browsers warn the user if the response was not signed in the first place or if the signature could not be confirmed. This is usually called that the “connection is insecure", “a secure connection could not be established” or something similar. You should never send passwords, credit card information or any other sensitive information if you get any of those warnings or if the URL starts with http:// instead of https://.
One-way hashing
This type of encryption is quite different and we do not use it to encrypt our data, as anything encrypted with “one-way hashing” - or simply “hashed” - is impossible to decrypt, or un-hash. If we were to hash our data and save only the output but not the original data, then we will have lost that data forever. While it is never a method to encrypt data, it is still useful to understand one-way hashing to stay safe in the digital world as it is central to all secure password handling.
While it is impossible to un-hash anything, the same piece of information is always hashed the same way, i.e. the output or the “hash” is the always the same, as long as the input is the same and the same hashing algorithm is used. So, if the input to a hashing algorithm was the same, then the output is always the same. This also means that if the output to a hashing algorithm is different, then the inputs used could never have been the same.
Example of use-case of one-way hashing
When you create an account on any service like Facebook.com or DropBox.com they take your password and use it in an one-way hashing algorithm, and then only saves the output in their data base. If their servers would be hacked, or if their data base engineers were undercover hackers, all they would see is the output of the one-way hash algorithm and not the input, i.e. your password. Every time you log in to your account, your password will be put into the same hashing algorithm and the output will be compared to what they saved in their data base when you created your account. If the outputs or hashes are the same, then they know that you used the same password both times, even though they never saved your password when you signed up.
This is not difficult to implement, so any service that does not hash user passwords before saving them is extremely insecure and should always be avoided. No web-company will ever show you their database but here are examples on how you can know that they did not hash your password before storing it. If you ever encounter any of the scenarios below, stop using that service and report it on cyber-security forums immediately.
- They are able to send you the password if you forget it. Instead they send you a temporary password or a password rest link
- Anyone in customer support is able to tell you your password, or they say that they can see it on their screen
Example of weaknesses of one-way hashing
Common input leads to common output. Very common passwords like “Password1”, “qwerty” or “12345678” have very well known hashes, and hackers have already calculated and published the hashes for hundred of thousands of passwords. So if someone hacks into a database and you have used “12345678” as your password, then your hash is quickly identified as a known hash, and the hacker immediately knows your password. In addition to common long-ish passwords like “Password1”, hackers has already calculated all hashes for all possible combinations of passwords up to a few letters long. That is why services require you to have password of a certain lengths, and use symbols and numbers, as combination of only letters is the first thing hackers calculate as they cover all combinations of longer and longer passwords.
Encryption in Transit and at Rest
Encryption in Transit
Encryption in transit is by far the most important type of encryption. Service providers almost always take care of (i.e. Survey Solutions, SurveyCTO, OneDrive). However, when using less established services, confirm that they use encryption in transit by looking at the internet address. If the service provider uses secure transfer methods, you will see https:// in the internet address instead of simply http://. Data sent from an API using http:// can easily be spied on.
Never send anything of importance over the internet unless the URL starts with https://. Data transferred over an http:// connection can often be openly read by every server through which the data passes. Those servers are controlled by governments and private companies; hackers can easily tap into their traffic and read data, copy files, read passwords etc. https:// is not the only secure way to transfer data over the internet, but it is the one that researchers should know of as we use it frequently. If setting up advanced protocols to send files, make sure that they are set up to be secure. For example, instead of using FTP, use FTPS.
Note that encryption in transit has nothing to do with a service requiring a username or password. While a password-protected resource can only be requested by someone with the correct password, once it is in transit to the authorized user, the servers handling it can still see it if it is not encrypted.
Encryption at Rest
Encryption at rest means that the data stored on the server is scrambled in such a way that it is unreadable by anyone – even if they can access the file directly. If the data is not encrypted at rest, then anyone with access to the server can read that data (including, for example, the host company and your team’s administrators). If the data is encrypted at rest, however, the data is impossible to read even if someone gained unauthorized access to the database or the files where the data is stored. Encryption at rest uses an authorization tool called a public/private key pair.
Encryption at rest is not as easy to implement as encryption in transit. Both types of encryption use a private/public key pair to ensure that no unauthorized person gets access to the data. Since the time during which the data must be encrypted in transit is so short, the web servers never need to give users the key pair. Once the transfer is complete, the key pair is discarded and never needed or used again. However, in encryption at rest, the research team encrypts the data at one point in time and will access the data at a later point in time. The computer therefore must give the key pair to a human. As is so often the case, the weakest link is always the human factor. No private/public key pair is secure if the computer that generated the key saves it or is able to re-generate it. Thus, humans must safekeep the key pair. If we lose it, the data is lost forever and there is no way whatsoever to regain access to it.
The exact way in which encryption at rest is implemented depends on the service. For more information, read the instruction specific to your service.
Public/Private Key Pair
The following principles always hold for public/private key pairs:
- The key pair can only be created once. Any services that claim to be able to re-generate the private key for you are not safe: this is equivalent to them having your password. This either means that the data is not properly encrypted or that they saved a copy of your private key that gives them full access to your encrypted data.
- Most services provide a convenient way of decrypting your data, but if you are not asked for the key every time the data is viewed or decrypted, then your data is not encrypted properly.
Password and Key Management
When using public/private key pairs, you need to keep a lot of keys safe and well-organized. You can never store, organize, or transfer them using unsecure methods like email or Dropbox. DIME Analytics recommends using password managers as a convenient and safe solution for storing private keys and passwords. While there are many password managers out there, DIME Analytics recommends LastPass, whose free, basic tier satisfies all requirements a research team would ever need.
Password managers like LastPass allow you to store small text files or string fields, as private/public key pairs sometimes come in file format. LastPass can save keys as small text files or as strings for you as secure notes, which you can also use for other important information, like noting to which encryption container the private key pertains. Password managers like LastPass can also help you randomize passwords so that your passwords are impossible-to-guess, long strings. Since the password manager store and remembers this password for you, you will not need to memorize it and you can easily have a different long random string for each account you have.
Some password managers allow you to share passwords across accounts. This is a great feature: if one person updates the password, then everyone else has the updated password in their account immediately. Password sharing is sometimes a paid feature. One way around paying is for the research team to create one account and share access to that account. However, make sure to not share this account to more people than those who really needs it.
Encryption with SurveyCTO Data
When programming questionnaires in SurveyCTO, use the key generator to create a public-private key pair for your data with the name of your intended survey. These will download two files to your laptop: these are the keys. Create an encrypted survey with the public key. Then put the keys in the vault via LastPass or VeraCrypt. Note that you will need the private key for the form data download. Store encrypted data iefolder’s EncryptedData folder. Note, however, that while iefolder makes the folder, it does not encrypt it.
In your first cleaning do-file, de-identify the PII data to create a de-identified dataset. If you’re using Veracrypt, the veracrypt command allows Stata to call for VeraCrypt to mount the drive. Then you have to manually enter the password anytime you run the do-file. The first cleaning file should move the non-PII version of your data to the regular data folder.
Back to Parent
This article is part of the topic Data Security
Additional Research
- DIME Analytics' slides on Encryption
- Poverty Action Lab’s Data Security Procedures for Researchers