Offline-first app: How bad is it to build one
More than just cache, an architectural decision!
Concept
Your client needs an app that functions fully or partially without an internet connection. This means as many features as possible should be available offline.
Examine the app and define offline behaviors for every component. Deal with interrupted communication. Planning the synchronization of data. This focus on supporting dysfunctional communication systems is what we call “offline-first”.
Then, once you have finished planning and implementing the features for the lack of internet connection, you will have an “offline-first app”!
From scratch to action
To be fair… “An app that supports offline functionality” is a vague requirement, and that’s okay!
No technique can resolve all your problems, but using some of them to create a custom solution is the right direction.
You have to think here! I’ll show you some approaches, and you'll use/mix some of them to build the proper solution.
Our app will be a simple note app. It has the Flutter code, but also a Go API. Due to the lack of public APIs with PUT and POST operations, I had to create my own API.
Store data
Without an internet connection, our best guess is to rely on stored information. As communication stabilizes, the local data can be synchronized.
Cache queries
The simplest and most well-known technique to keep the app functional when lost communication.
Whenever the server responds to a GET request, you should store and use the data if communication is lost.
Two issues must be addressed: When to copy and when to use the stored data.
Copy the data just after receiving a response, you may also add the date and time you received it to know how old the data is.
We can rely on the local data saved earlier when a request fails because of the lack of an internet connection. Decide if caching the responses will be viable for other types of failures.
To avoid using too old data, lifetime information can be defined and stored with the response.
Let’s practice! We can use an interceptor in our HTTP client to store the data. In this case, I’m using Dio as an HTTP client.
class CacheQueryInterceptor extends Interceptor {
final LocalStorageService _localStorage;
const CacheQueryInterceptor(this._localStorage);
bool _hasNoConnection(DioException err) {
return err.response == null;
}
@override
void onResponse(
Response response,
ResponseInterceptorHandler handler,
) async {
// Store response for GETs
if (response.requestOptions.method == "GET") {
final storableRequest = StorableRequest.fromDio(response.requestOptions);
await _localStorage.save(storableRequest.toStorableString(), jsonEncode(response.data));
}
handler.next(response);
}
@override
void onError(
DioException err,
ErrorInterceptorHandler handler,
) async {
// If GETs fails, we retrieve the previously stored response
if (_hasNoConnection(err) && err.requestOptions.method == "GET") {
final request = err.requestOptions;
final storableRequest = StorableRequest.fromDio(err.requestOptions);
final storedResponse = await _localStorage.get(storableRequest.toStorableString());
if (storedResponse != null) {
handler.resolve(Response(requestOptions: request, data: jsonDecode(storedResponse)));
return;
}
}
handler.next(err);
}
}
Request queue
When dealing with other request methods like POST, PUT, PATCH, DELETE, etc. The process becomes more complex.
We need to store the request before sending it and, in case it fails… Put it on a pending queue, then resend it at some point. We will discuss synchronization strategies further.
Create a synchronization class or function that can be invoked as needed. It will be responsible for resending the pending requests.
For synchronization’s decision support you should add information to the stored requests like number of attempts, creation date, and any other thing you believe is useful.
Watch out for the requests sequence! We can’t delete what was not created or edit something that was deleted.
class RequestQueueInterceptor extends Interceptor {
static const String kListKey = 'request-queue';
final LocalStorageService _localStorage;
const RequestQueueInterceptor(this._localStorage);
bool _hasNoConnection(DioException err) {
return err.response == null;
}
@override
void onResponse(
Response response,
ResponseInterceptorHandler handler,
) async {
// Remove the request from the queue
if (response.requestOptions.method != "GET") {
final storableRequest = StorableRequest.fromDio(response.requestOptions);
await _localStorage.removeFromList(kListKey, storableRequest.toStorableString());
}
handler.next(response);
}
@override
void onError(
DioException err,
ErrorInterceptorHandler handler,
) async {
// Insert a request in the pendency queue if it fails
if (_hasNoConnection(err) && err.requestOptions.method != "GET") {
final storableRequest = StorableRequest.fromDio(err.requestOptions);
final list = (await _localStorage.getList(kListKey)).map((e) => StorableRequest.fromStorableString(e)).toList();
if (!list.contains(storableRequest)) {
await _localStorage.addInList(kListKey, storableRequest.toStorableString());
}
}
handler.next(err);
}
}
Local database
In place of cache HTTP requests, another approach is to have a local database that our app will consume directly.
The first step is to build a model of what you want to save. For our note app, we have a model called Note and a strategy to transform it into a Map so we can store and retrieve it.
class Note extends Synchronizable {
final String id;
final String content;
final DateTime createdAt;
const Note({
required this.id,
required this.content,
required this.createdAt,
super.isSync = false,
super.isDeleted = false,
});
factory Note.fromMap(Map<String, dynamic> map, {bool isSync = false}) {
return Note(
id: map['id'] is num ? map['id'].toString() : map['id'],
content: map['content'],
createdAt: DateTime.parse(map['createdAt']),
isSync: map['isSync'] ?? isSync,
isDeleted: map['isDeleted'] ?? false,
);
}
Map<String, dynamic> toMap() {
return {
'id': id,
'content': content,
'createdAt': createdAt.toIso8601String(),
'isSync': isSync,
'isDeleted': isDeleted,
};
}
}
We also should add some useful data to our models. For example, if it is synced, last sync date, etc.
Also if requested to delete a model, just remove it from the local db when you have guaranteed it is synchronized. Otherwise, is smart to have a will_be_deleted variable.
A key advantage of this solution is providing visual feedback for the differences between local and remote data.
A synchronization entity is also necessary. Our domain rules will only know the local database, and content will be updated through the remote API when appropriate.
The class to manage and update the local database will be the NoteRepository. It works like the previous strategies, but when offline, our local list of notes will be updated with the isSync property set as false.
class NoteRepository extends Repository {
final ApiDatasource _datasource;
final NotesCache _cache;
const NoteRepository(this._datasource, this._cache);
Future<List<Note>> getAllNotes() async {
// Try to sync
try {
final response = await _datasource.httpClient.get('/notes');
final notes = (response.data as List).map((e) => Note.fromMap(e, isSync: true)).toList();
await _cache.saveNotes(notes);
} catch (e) {
// handle the error
}
// retrieve the local data
return _cache.getNotes();
}
Future<Note> getById(String id) async {
// Try to sync
try {
final response = await _datasource.httpClient.get('/notes/$id');
final note = Note.fromMap(response.data);
final existingNote = await _cache.getById(id);
if (existingNote != null) {
await _cache.editNote(note);
} else {
await _cache.saveNote(note);
}
} catch (e) {
// handle the error
}
// retrieve the local data
final note = await _cache.getById(id);
if (note != null) {
return note;
} else {
throw Exception('Note not found');
}
}
Future<Note> createNote(Note note) async {
// Save locally first
Note currentNote = note.copyWith(isSync: false);
await _cache.saveNote(currentNote);
// Try to sync
try {
final response = await _datasource.httpClient.post('/notes', data: {'content': note.content});
currentNote = Note.fromMap(response.data, isSync: true);
await _cache.editNote(currentNote);
} catch (e) {
// handle the error
}
// retrieve the local data
return currentNote;
}
Future<void> editNote(Note note) async {
// Save locally first
await _cache.editNote(note.copyWith(isSync: false));
// Try to sync
try {
await _datasource.httpClient.put('/notes/${note.id}', data: {"content": note.content});
await _cache.editNote(note.copyWith(isSync: true));
} catch (e) {
// handle the error
}
// has nothing to retrieve
}
Future<void> deleteNote(Note note) async {
// Save locally first
await _cache.editNote(note.copyWith(isSync: false, isDeleted: true));
// Try to sync
try {
await _datasource.httpClient.delete('/notes/${note.id}');
await _cache.deleteNote(note);
} catch (e) {
// handle the error
}
// has nothing to retrieve
}
@override
Future<void> sync() async {
// Create a copy of the list
final allNotes = List<Note>.from(await _cache.getNotes());
// Iterate to sync every unsync model
for (Note note in allNotes) {
if (note.isSync) {
continue;
}
try {
if (note.isDeleted) {
if (note.hasTempId) {
_cache.deleteNote(note);
} else {
await deleteNote(note);
}
} else if (note.hasTempId) {
await createNote(note);
} else {
await editNote(note);
}
} catch (e) {
continue;
}
}
}
}
Synchronization
We called our API to create a model and it failed, now our app is unsynchronized. How to fix it? When should we retry to make the requests? Which options are there?
You already have one version of a sync function in the NoteRepository, now focus on how/when to use it.
Ask the user
The simplest approach is to let the user know that something was not saved on the server. So we present him with an option to start the synchronization process.
A friendly and transparent option that permits us to pass the responsibility of when the synchronization should take place.
Period timer
We can set our app to sync any pending request on a fixed period. In this case, this process doesn’t need to be transparent to the user, he/she won’t know that some process is running.
A sacrifice of processing power for more tries aiming to diminish the time spent in an unsynchronized state.
If you are not familiar with the Timer class in Flutter and how to guarantee its behavior with automated tests, check out this other article.
Lazy sync
Important note: There is no need to synchronize all data at once. We can group the requests and shoot a try every time the user needs that specific data.
With the pending data structured and organized, we can block access to determined features if the synchronization does not succeed (again).
Conflict resolution
While the app can manipulate our model, the server or other user can also change the values of the same model resulting in a conflict situation.
Being aware of those possibilities and defining a path to find the final result of these concurrent changes is to work on conflict resolutions.
Last change wins
The name explains itself! Attach a timestamp to the model every time you make a change, when a conflict happens we can discard the oldest one.
It is easy to implement, but be aware that some important information is being lost. Not always a choosable option.
Thought path
We can check which properties of the model have changed, and use both new values if different properties were updated.
But it costs a huge effort. We could use a timestamp for each property or a detailed log for every change event.
Priority level
Depending on the change type, we can choose who rules. If you have a delete-type request against an update-type request. Maybe we can just delete it!
Or a change made by an administrator against a topical user, So the system administrator has the preference.
For our example app, the user-side data always won! Because I’m the only one using it. =)
Last tips
Encrypt the stored data
Store app content in the user's phone/PC is a security issue. Encrypting and obfuscating local data is a smart practice.
Be aware of versioning conflicts
As our app evolves, the system versions are updated and the data format can change.
Handle the data transformation process smoothly. An option is to set your app to be blocked until the user updates to a minimum version.
Keep optimizing
- Sync only when there is a Wi-fi connection (suggestion).
- If files are being stored locally, search for algorithms to reduce their size and make comparisons.
- Background processment to minimize the user experience.
- Limit the amount of data to synchronize per period, especially if the user has low battery power.
Conclusion
Offline-first apps need a quite amount of effort to implement and maintain. It’s primordial to have validated and clear requisites for the right choices can be made. Hopefully, just caching the queries will be enough!
More approaches and techniques about this topic are out there to be explored, do not limit yourself to the options presented here.
Despite having an example project, The concepts are what I wanted to focus on and share. Since some techniques imply architectural changes and the complexity of asynchronous data handling, the wrong decisions can be devastating… be smart!
Example project
Although I can’t write a code for every approach discussed in this article, I hope this simple example can help clarify your understanding.
I did struggle this time! Still, I took the opportunity to learn the Go language, so don’t expect too much from the code.
Thanks for reading! If you want to read more of my thoughts and programming tips, then check out my articles and follow or subscribe for upcoming content.