DataStore sync errors and sync delay on iOS on version 2.4.0

[filipesbragio]

Description

Likely related to #5217.

Using Amplify version 2.4.0 on iOS. Turning off connectivity(wifi/mobile) will cause a SocketException and cause data sync to stop. It seems like the retries use exponential backoff, every time it fails it takes around twice as much until it tries again. DataStore will only re-establish data synchronization when this retry is attempted and mobile has connectivity again, it does not try to connect when connectivity is re-established.

This causes scenarios such as the following: If the user is offline and several retries occur then the next retry can be dozens of minutes in the future. If the user becomes online the data will not be synced until that retry is executed. If the next retry is in 20 minutes and the mobile connects to the server, he can use the app for up to 19 minutes and no data will be synced. The server will only receive that users data when he re-opens the app with connectivity or when a retry runs while he is online.

Minimizing the app does not change this, only closing and reopening the app.

This bug only happens on iOS. We didn't have any errors on Android and the sync was done as soon as connectivity was re-established.

Here is the error message, stack trace and the periods in which the app retried.

SocketException : Failed host lookup: 'xxxxxxxxxxxxxxxxxxxxxx.appsync-realtime-api.us-east-2.amazonaws.com'

OS Error: nodename nor servname provided, or not known, errno = 8

0 _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:721:35)
1 _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1914:26)
2 RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
3 RawSecureSocket.startConnect (dart:io/secure_socket.dart:299:22)
4 SecureSocket.startConnect (dart:io/secure_socket.dart:77:28)
5 _ConnectionTarget.connect (dart:_http/http_impl.dart:2480:26)
6 _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2923:12)
7 _HttpClient._getConnection (dart:_http/http_impl.dart:2928:12)
8 _HttpClient._openUrl (dart:_http/http_impl.dart:2783:12)
9 _HttpClient.openUrl (dart:_http/http_impl.dart:2620:7)
10 _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1021:42)
11 WebSocket.connect (dart:_http/websocket.dart:320:22)
12 new IOWebSocketChannel.connect (package:web_socket_channel/io.dart:81:28)
13 connect (package:web_socket_channel/src/_connect_io.dart:15:24)
14 new WebSocketChannel.connect (package:web_socket_channel/src/channel.dart:144:16)
15 AmplifyWebSocketService._init (package:amplify_api_dart/src/graphql/web_socket/services/web_socket_service.dart:81:40)

These are the times when each retry was executed, I stopped registering them after the 27 minute delay.
13:42:41
13:42:42
13:42:46
13:42:52
13:43:05
13:43:31
13:44:22
13:46:05
13:49:30
13:56:19
14:10:00
14:37:19

Categories

• Analytics
• API (REST)
• API (GraphQL)
• Auth
• Authenticator
• DataStore
• Notifications (Push)
• Storage

Steps to Reproduce

See description

Screenshots

No response

Platforms

• iOS
• Android
• Web
• macOS
• Windows
• Linux

Flutter Version

3.22.3

Amplify Flutter Version

2.4.0

Deployment Method

Amplify CLI

Schema

No response

[NikaHsn]

Sorry that you are facing this issue and thanks for reporting it. We will look into this and get back to you when we have updates.

[filipesbragio]

Thanks for the swift response, let me know if you need any additional information.

[giulitu95]

I have exactly the same problem. This is a very limiting problem because every time datastore tries to establish the connection and fails, we cannot catch and handle the SocketException.
Furthermore, if I try to force the datastore to restart synching with Amplify.Datastore.start(), the synchronization does not start. As @filipesbragio mentioned, to immediately synchronize datastore after a relatively long time the phone is offline (e.g. 10 minutes), the only way is to restart the application.

In the time interval when the device is online and amplify has not already detected such networkStatus event (because of exponential backoff), datastore modifications done while the device was offline are not pushed to AppSync while subsequent modifications are correctly synchronized. Note that the behavior is very similar to the one described in the #5217:

Any changes we make while offline seem to be ignored and never updated on dynamoDB. Any changes made after becoming online again sync normally.

(It could be also related to #5139)

[stam0912]

I have exactly the same problem. This is a very limiting problem because every time datastore tries to establish the connection and fails, we cannot catch and handle the SocketException. Furthermore, if I try to force the datastore to restart synching with Amplify.Datastore.start(), the synchronization does not start. As @filipesbragio mentioned, to immediately synchronize datastore after a relatively long time the phone is offline (e.g. 10 minutes), the only way is to restart the application.

In the time interval when the device is online and amplify has not already detected such networkStatus event (because of exponential backoff), datastore modifications done while the device was offline are not pushed to AppSync while subsequent modifications are correctly synchronized. Note that the behavior is very similar to the one described in the #5217:

Any changes we make while offline seem to be ignored and never updated on dynamoDB. Any changes made after becoming online again sync normally.

(It could be also related to #5139)

I observed the following behaviour. In a normal situation, if the internet connection is gone, there should be reconnect attempts (8 attempts in 50s by default), the websocket should reconnect as soon as the internet is back. However, in a buggy situation, it doesn't even attempt to retry or close the socket properly. The web socket connection doesn't recover even if the internet is back. The web socket will be timeout in 5 minutes. And within this 5 minutes, it won't let you do anything. You can't call Amplify.DataStore.stop()/start(). And finally after 5 minutes, the Amplify.DataStore.stop()/start() work again.

Something I've tried to make it reconnect automatically (but not 100% working):

1. Listen to the Api hub event and manually restart the DataStore. Losing internet connection will give you a FAILED status

  void observeApiHubEvent() {
    Amplify.Hub.listen(HubChannel.Api, (ApiHubEvent event) async {
      if (event is SubscriptionHubEvent) {
        print("Amplify API status: ${event.status}");
        if (event.status == SubscriptionStatus.failed) {
          if (Platform.isAndroid) {
            await Amplify.DataStore.stop();
            await Future.delayed(const Duration(seconds: 5));
            await Amplify.DataStore.start();
          }
        }
      } else {
        print("Amplify API event: ${event.eventName}");
        print("Payload: ${event.payload.toString()}");
      }
    });
  }

Issue: Sometimes it works, sometimes it doesn't, especially when it is in release mode.

2. Set a 5 timer and restart the datastore after internet interruption.
   I use the connectivity_plus package to monitor the internet connection

  void startMonitoring() {
    print("Start monitoring internet connection");

    _subscription = Connectivity()
        .onConnectivityChanged
        .listen((List<ConnectivityResult> result) async {
      // Received changes in available connectivity types!
      print('Connectivity changed (stream): $result');
      if (result.contains(ConnectivityResult.mobile) ||
          result.contains(ConnectivityResult.wifi) ||
          result.contains(ConnectivityResult.ethernet)) {
        isInternetConnected = true;
        try {
          await Amplify.DataStore.stop();
          await Amplify.DataStore.start();
        } catch (error) {
          print("DataStore restart error: $error");
        }
      } else {
        isInternetConnected = false;
        if (dataStoreRestartTimer != null) {
          dataStoreRestartTimer!.cancel();
        }
        //restart datastore after 5 minutes
        dataStoreRestartTimer = Timer(const Duration(minutes: 5), () async {
          //fall back datastore restart after internet interruption
          try {
            await Amplify.DataStore.stop();
            await Amplify.DataStore.start();
          } catch (error) {
            print("DataStore restart error: $error");
          }
        });
      }
    });
  }

Issue: Works only if app is in foreground or you enable your app to run in the background

These are just my ideas, I need help to think of a workaround.

[Equartey]

Hi @stam0912, can you try 2.4.1? It should resolve the behavior when calling Amplify.DataStore.Stop() in this scenario.

Web socket issues when the app is in the background in release mode has been reported, you can track that here: #5041

As for reconnecting during retry/backoff timeouts, this has been reproduced. However, we are still investigating how to best resolve it.

We will use this issue for updates regarding retry/backoff logic. All other issues should be tracked separately.

[giulitu95]

Is there a workaround to catch WebSocket exceptions every time amplify retries to reconnect?

[Equartey]

Hi @giulitu95, the current best option available is to monitor the Hub events channel as described in this docs page, #subscription-connection-status.

[giulitu95]

Hi @giulitu95, the current best option available is to monitor the Hub events channel as described in this docs page, #subscription-connection-status.

Thank you @Equartey, actually I don't need to catch the event but I would like to avoid that the library raises a SocketException which I cannot handle, every time the internet connection is checked. Firstly because since typically in development mode the editor is set in a way that the execution is paused whenever an uncaught exception is thrown (this could be annoying while debugging the application) and second because by using error tracking services like sentry, every time this exception is thrown a new error is reported by the service (but is not something that we want to actually track).
Is this a wanted behavior or is something that in the next releases will be fixed?

[ekjotmultani]

Hey @giulitu95, apologies for the inconvenience caused by this error being thrown when you don't want to track it! Currently this is the expected behaviour, however we will investigate if there is a way to catch the exception silently.