How to stream an encrypted backup of the entire device to remote host?


Question

I have an Android device that has no free space and no SD-card that I can replace (thanks to OnePlus policy). I also have no free space on PC to accommodate the backup so I wanted to upload it to the cloud. I have SSH and root access to the device. I also have a remote FTP server space (that I not trust, though).



I would like to make a password protected tar or zip file and upload it there. I'm not sure if a zip of / is sufficient for backup purposes.



The problem is there is no free space left on the device so I need to create the zip file on the fly and "stream" it to the backup server (it's SFTP on port 30).



I checked that the SSH, rsync and zip commands are available but I'm not sure how to do a full backup, encrypted zip compression and SFTP upload without writing to the disk.



How can I combine commands to do that while connected to the device over SSH?


Answer

We have authenticated login, encrypted archive, rsync and multiple transport protocols (including HTTP, WebDAV, FTP and SSH/SFTP) in one tool: rclone. It supports local filesystem and a number of cloud storage. Additionally it does have all of the four mentioned servers builtin. It means that all you need on both sides is rclone. No root access is necessary either. If you want to mount remote directory (using rclone mount), however, that requires FUSE support and root access on Android.



rclone does have builtin encryption i.e. files on remote will be created encrypted. Also there are multiple authentication methods supported including username/password and RSA keys (with SFTP), but those are usually unnecessary when encryption is already there.



A simple use case:



On server:



Run minimal SSH server if not already running:



~$ mkdir -p BackupDir/Encrypted
~$ rclone serve sftp --user <username> --pass <password> --addr :2222 BackupDir


On phone:



Create obscure password:



~$ rclone obscure <password>


Create configuration file (use rclone config for simple steps):



# rclone.conf

[PLAIN_TARGET]
type = sftp
host = <server_ip>
port = 2222
user = <username>
pass = <obscure_password>
#key_file = /path/to/.ssh/id_rsa

[CRYPT_TARGET]
type = crypt
remote = PLAIN_TARGET:/Encrypted
filename_encryption = standard
directory_name_encryption = true
password = <obscure_password>


Copy test file to server without and with encryption:



~$ echo Hi >test_file
~$ rclone --config rclone.conf sync test_file PLAIN_TARGET:/
~$ rclone --config rclone.conf sync test_file CRYPT_TARGET:/


SFTP supports modification timestamps, WebDAV and FTP both don't. However symlinks cannot be transferred as symlinks, those are either ignored or original file is copied or .rclonelink file is created. A way to copy symlinks and all metadata including timestamps, extended attributes (like SELinux labels), and ACLs is to throw tar stream at rclone rcat:



~$ tar --xattrs -cpvzf - test_file | rclone rcat CRYPT_TARGET:/test_file.tgz


Similarly you can get back the file using rclone cat. This will also largely reduce transfer time (even up to less than half) particularly if you have large number of small-sized files. Adding reasonable compression to tar may even reduce space usage and time, but high compression can go wrong too. However this way incremental sync won't work and you need to untar to use files on remote. It's also possible (by some scripting) to find only changed files first by running rclone check or by passing option --dry-run to rclone sync command and then add those files to tar archive.




I'm not sure if a zip of / is sufficient for backup purposes.




On Android rootfs / contains a number of directories and mount points - pseudo filesystems like sysfs, procfs and read-only filesystems like system and vendor partitions - which cannot be or don't need to be backed up. User settings, apps data and personal data is stored in /data partition. For details see Android Partitions and Filesystems and How disk space is used on Android device?.



So you can backup selected directories from /data or the whole partition at maximum. However, as mentioned by @alecxs in comment, if you intend to restore apps data (in /data/data), you'll have to minutely take care of files metadata - in particular UIDs/GIDs and SELinux contexts.



Say you want to backup whole /data partition, do:



~# tar --xattrs -cpvzf - /data | rclone -P rcat PLAIN_TARGET:/data.tgz


For ideas on how to dump complete partitions (block devices), see How to recover a deleted file from /data partition?


Topics


2D Engines   3D Engines   9-Patch   Action Bars   Activities   ADB   Advertisements   Analytics   Animations   ANR   AOP   API   APK   APT   Architecture   Audio   Autocomplete   Background Processing   Backward Compatibility   Badges   Bar Codes   Benchmarking   Bitmaps   Bluetooth   Blur Effects   Bread Crumbs   BRMS   Browser Extensions   Build Systems   Bundles   Buttons   Caching   Camera   Canvas   Cards   Carousels   Changelog   Checkboxes   Cloud Storages   Color Analysis   Color Pickers   Colors   Comet/Push   Compass Sensors   Conferences   Content Providers   Continuous Integration   Crash Reports   Credit Cards   Credits   CSV   Curl/Flip   Data Binding   Data Generators   Data Structures   Database   Database Browsers   Date &   Debugging   Decompilers   Deep Links   Dependency Injections   Design   Design Patterns   Dex   Dialogs   Distributed Computing   Distribution Platforms   Download Managers   Drawables   Emoji   Emulators   EPUB   Equalizers &   Event Buses   Exception Handling   Face Recognition   Feedback &   File System   File/Directory   Fingerprint   Floating Action   Fonts   Forms   Fragments   FRP   FSM   Functional Programming   Gamepads   Games   Geocaching   Gestures   GIF   Glow Pad   Gradle Plugins   Graphics   Grid Views   Highlighting   HTML   HTTP Mocking   Icons   IDE   IDE Plugins   Image Croppers   Image Loaders   Image Pickers   Image Processing   Image Views   Instrumentation   Intents   Job Schedulers   JSON   Keyboard   Kotlin   Layouts   Library Demos   List View   List Views   Localization   Location   Lock Patterns   Logcat   Logging   Mails   Maps   Markdown   Mathematics   Maven Plugins   MBaaS   Media   Menus   Messaging   MIME   Mobile Web   Native Image   Navigation   NDK   Networking   NFC   NoSQL   Number Pickers   OAuth   Object Mocking   OCR Engines   OpenGL   ORM   Other Pickers   Parallax List   Parcelables   Particle Systems   Password Inputs   PDF   Permissions   Physics Engines   Platforms   Plugin Frameworks   Preferences   Progress Indicators   ProGuard   Properties   Protocol Buffer   Pull To   Purchases   Push/Pull   QR Codes   Quick Return   Radio Buttons   Range Bars   Ratings   Recycler Views   Resources   REST   Ripple Effects   RSS   Screenshots   Scripting   Scroll Views   SDK   Search Inputs   Security   Sensors   Services   Showcase Views   Signatures   Sliding Panels   Snackbars   SOAP   Social Networks   Spannable   Spinners   Splash Screens   SSH   Static Analysis   Status Bars   Styling   SVG   System   Tags   Task Managers   TDD &   Template Engines   Testing   Testing Tools   Text Formatting   Text Views   Text Watchers   Text-to   Toasts   Toolkits For   Tools   Tooltips   Trainings   TV   Twitter   Updaters   USB   User Stories   Utils   Validation   Video   View Adapters   View Pagers   Views   Watch Face   Wearable Data   Wearables   Weather   Web Tools   Web Views   WebRTC   WebSockets   Wheel Widgets   Wi-Fi   Widgets   Windows   Wizards   XML   XMPP   YAML   ZIP Codes