Offline safe sync directory on sd card on non-rooted Android with Ubuntu PC


Question

Is there a way how to offline safely synchronize directories between sd card on Android which is not rooted and PC with Ubuntu?



Safely = using secured connection between the devices (for example using ssh).



Offline = without Internet access, but the PC and smartphone are connected through LAN.



Synchronize directories = a process which transfers data between smartphone and PC in a way that at the end of the process, both devices contain exactly same data in selected directories (preferably, but not necessarily, excluding filtered files). The process is able to detect deleted data so it doesn't copy file which was originally on both devices and later deleted on one of them.



My systems




  • Android 6.0.1 (not rooted)

  • Xubuntu 18.04 64 bit PC



What I already tried/checked (without success)



Unison





Unison for Android




  • https://play.google.com/store/apps/details?id=com.nerdysoftware.unison

  • Not secure - from application description: "No SSH access is necessary - clients connect directly via socket. Because Unison clients connect to your device over a direct socket connection instead of over SSH, the server should only be run when your Android device is connected to a trusted network, like a home WiFi. Running the server on other networks could expose files on your device to untrusted parties."

  • Also seems to use older version of Unison (this application requires exactly same versions on server and client to allow data transfer).



Rsync





SSHDroid




  • https://play.google.com/store/apps/details?id=berserker.android.apps.sshdroid

  • For using SSH with another software (see bellow)

  • Cannot write/modify SD card using the ssh connection. I went to settings -> Applications -> SSHDroid -> Permissions: storage is enabled. There I opened context menu -> all permissions: reading SD card available. Modification/deleting data on SD card available. In other words, I checked the application has permission to modify the SD card: it has.



SSHFS with SSHHelper





SSHFS with SimpleSSHD





What I already tried (with mixed/partial success)



Combination: Unison (on PC) + some SSH server (Android) + SSHFS (on PC)




  • In case I could find working SSH server; use SSHFS to mount remote Android device locally and then use Unison for synchronization.

  • Results:


    • When the SSH is actually rw, then it works, but it's so slow that it's impractical - it must ask for every single file and directory over the network. Of course, other apps (like Unison) must do that too, but they don't ask for every single file sequentially (not: client send request for single file - latency - server checks and sends response - client receive response and go to next file).

    • Unison requires permission to rename files; it always creates temporary files and directories and when everything is downloaded and double-checked, then old files (if any) are deleted and new files renamed. So if renaming is not allowed, it doesn't work.




SSH/SFTP Server - Terminal




  • https://play.google.com/store/apps/details?id=net.xnano.android.sshserver

  • This actually works but has two problems;


    • It requires entering password several (>10) times when mounted as sshfs (I tried trivial one and and made sure it's not typo) and doesn't support ssh keys. I was able to solve this problem with a hack: zenity --password | sshfs user@host:/ "$PWD/data" -o reconnect,password_stdin, but I'm not sure how secure it is.

    • This ssh is unbelievably slow. Much slower when compared with SSHDroid and SSHHelper (not only with sshfs, but also when I simply connect using ssh and terminal; typing and responses are really, really slow). Maximum speed is 2 MB/s, but most of the time it's ~500kB/s.

    • Cannot rename files (permission error) so it makes it unusable with Unison (on PC).




Syncthing




  • https://play.google.com/store/apps/details?id=com.nutomic.syncthingandroid

  • Problem: cannot modify SD card (on non-rooted systems) and devs are not going to fix this anytime soon.

  • The only solution is to use obscure, hard-to-reach deep directory structure: /storage/XXXX-XXXX/Android/data/com.nutomic.syncthingandroid/files.

  • But for this particular directory, it works. It's speed is about 5 - 10 MB/s and I suspect that it's being throttled down by SD card itself (or Android). For now, I'm using this solution, but the directory is really inconvenient.

  • Smaller problems:


    • It doesn't support following symlinks and until someone (other than current devs) implement it, it's not going to be supported.




What problem I'm trying to solve



I "just" want to synchronize data on file level (some pdfs, images, etc.) between my PC and Android smartphone. I use multiple devices (desktop, laptop, etc.) and use Unison (note: I do not insist on this app) to synchronize data between them. Because I switch environment quite offten, I need it to work on one button press and to be fast as possible (which, for example, Unison does greatly). Because I work on different environments, I need to cover my cases: sometimes I'm without Internet access. Other times I'm connected to untrusted network.



This is also reason why I cannot use cloud (like Dropbox): it's slower than two devices communicating directly and sometimes I'm without Internet connection so I cannot synchronize even when the devices are connected on same network.



The reason why I want to sync with SD card is simply because it has bigger capacity so my files fits.


Answer

At the end, I settled with Syncthing. So, how does it fulfil my requirements:


Safely


It uses encryption. I didn't analyse how secure it is, so this is more matter of trust, but at least it shouldn't send data unencrypted.


Offline


Works exceptionally well. Local LAN detection seems reliable and is able to connect clients directly. I use it also at work (Windows-to-Windows and Windows-to-Linux) to share data with co-workers without exposing them to 3rd-party companies, it's fast and reliable. (Tens of gigabytes are transferred pretty quickly, so it seems the only bottleneck is the local network.)


Synchronize directories


Works well, but at the end, I decided to synchronize two-ways only one subdirectory instead of all data. It seems like Android likes to modify my files without my consent (especially media files), so except certain working directory, I block any changes done by the Android. It's inconvenient, but it's not issue with Syncthing.


Another problem is when synchronizing big files (thousands of megabytes and more), but again, it seems like issue with my device, but after a few system crashes, Syncthing eventually manages to transfer the data. (Later edit: it seems this issue really happens only on my Android device. Syncing single ~25GB file with a co-worked, Windows-to-Windows was without any problems.)


Pros



  • Reliable across platforms.

  • You just configure it and it works in background. If you're used, for example, to Dropbox, it's similar, except you don't share your data with 3rd party company. And it's actually faster (ymmv), because it synchronizes data between clients directly, not with server.

  • The application can be configured through browser. You can enable access from outer device (I recommend to enable password in this case!!!) which allows you to configure Android Syncthing through your desktop browser (you just need to find out IP address of your phone), which is really convenient.


Cons



  • As already mentioned in question, only directory on SD card, which can be synced, is /storage/XXXX-XXXX/Android/data/com.nutomic.syncthingandroid/files. It's really inconvenient, but I got used to it, somehow.

  • Linux desktop version doesn't handle symlinks. I solved it by using mount bindfs --resolve-symlinks --no-allow-other "srcDir" "shareDir".


Notes


I miss option to review data changes before they're synchronized, but this is matter of preference and application design, not an issue. And as stated in question, I couldn't find reliable Unison implementation on Android.


Topics


2D Engines   3D Engines   9-Patch   Action Bars   Activities   ADB   Advertisements   Analytics   Animations   ANR   AOP   API   APK   APT   Architecture   Audio   Autocomplete   Background Processing   Backward Compatibility   Badges   Bar Codes   Benchmarking   Bitmaps   Bluetooth   Blur Effects   Bread Crumbs   BRMS   Browser Extensions   Build Systems   Bundles   Buttons   Caching   Camera   Canvas   Cards   Carousels   Changelog   Checkboxes   Cloud Storages   Color Analysis   Color Pickers   Colors   Comet/Push   Compass Sensors   Conferences   Content Providers   Continuous Integration   Crash Reports   Credit Cards   Credits   CSV   Curl/Flip   Data Binding   Data Generators   Data Structures   Database   Database Browsers   Date &   Debugging   Decompilers   Deep Links   Dependency Injections   Design   Design Patterns   Dex   Dialogs   Distributed Computing   Distribution Platforms   Download Managers   Drawables   Emoji   Emulators   EPUB   Equalizers &   Event Buses   Exception Handling   Face Recognition   Feedback &   File System   File/Directory   Fingerprint   Floating Action   Fonts   Forms   Fragments   FRP   FSM   Functional Programming   Gamepads   Games   Geocaching   Gestures   GIF   Glow Pad   Gradle Plugins   Graphics   Grid Views   Highlighting   HTML   HTTP Mocking   Icons   IDE   IDE Plugins   Image Croppers   Image Loaders   Image Pickers   Image Processing   Image Views   Instrumentation   Intents   Job Schedulers   JSON   Keyboard   Kotlin   Layouts   Library Demos   List View   List Views   Localization   Location   Lock Patterns   Logcat   Logging   Mails   Maps   Markdown   Mathematics   Maven Plugins   MBaaS   Media   Menus   Messaging   MIME   Mobile Web   Native Image   Navigation   NDK   Networking   NFC   NoSQL   Number Pickers   OAuth   Object Mocking   OCR Engines   OpenGL   ORM   Other Pickers   Parallax List   Parcelables   Particle Systems   Password Inputs   PDF   Permissions   Physics Engines   Platforms   Plugin Frameworks   Preferences   Progress Indicators   ProGuard   Properties   Protocol Buffer   Pull To   Purchases   Push/Pull   QR Codes   Quick Return   Radio Buttons   Range Bars   Ratings   Recycler Views   Resources   REST   Ripple Effects   RSS   Screenshots   Scripting   Scroll Views   SDK   Search Inputs   Security   Sensors   Services   Showcase Views   Signatures   Sliding Panels   Snackbars   SOAP   Social Networks   Spannable   Spinners   Splash Screens   SSH   Static Analysis   Status Bars   Styling   SVG   System   Tags   Task Managers   TDD &   Template Engines   Testing   Testing Tools   Text Formatting   Text Views   Text Watchers   Text-to   Toasts   Toolkits For   Tools   Tooltips   Trainings   TV   Twitter   Updaters   USB   User Stories   Utils   Validation   Video   View Adapters   View Pagers   Views   Watch Face   Wearable Data   Wearables   Weather   Web Tools   Web Views   WebRTC   WebSockets   Wheel Widgets   Wi-Fi   Widgets   Windows   Wizards   XML   XMPP   YAML   ZIP Codes