Next Spaceship

Driving into future…

Caveat: Use Ubiquiti ER-X in China

| Comments

After reading some articles about Ubiquiti EdegeRouter X, I realize this is a device I have no resistance not to buy it. Gigabyte Ethernet, multi-WAN, configurable by CLI (thus a lot of fun), reasonable price, and most importantly, high reliability. I enjoy exploring configurations, and in the meantime, it’s a double-edge knife, if not configured well, it could have a very negative effect on your network performance. In this article, I will describe 2 configurations that could severely affect the performance when using it in China.

Ubiquiti has a very user-friendly forum for beginners to learn almost all technologies for EdgeRouter X. However, for the second issue (if you think it’s an issue), I don’t find anything on the internet. Probably it’s not that popular in China? Anyway, here is the story.

Requirement

  • Dual WAN, both are connecting through PPPoE. eth0 connects pppoe0 (WAN0), and eth1 connects pppoe1 (WAN1).

  • Six machines are in the LAN, with two connecting to WAN1 and four connecting to WAN0. All other devices will go through a load balancer to access the internet.

  • No VLAN, because I want all devices to connect directly freely.

Caveats

The first configuration is an easy one, the DNS name server. For dual WAN, we should use a name server at the system level instead of using the one automatically retrieved from ISP. Commands:

1
2
set interfaces ethernet eth0 pppoe 0 name-server none
set system name-server 114.114.114.114

(Reference: https://community.ui.com/questions/PPPoE-and-DNS-issues/c6ea0bb1-9a29-45c4-9aa5-eff94ef9f65b)

The second issue is not so obvious, and I find this issue by reading the message logs in /var/log/messages, sometimes, the load balance became inactive and then after a while became active again, and this happened once a while repeatly.

1
$ tail /var/log/messages
1
2
3
4
5
6
7
8
Feb 13 08:55:38 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe0 reachability changes to unreachable.
Feb 13 08:55:38 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe0 state changes to inactive.
Feb 13 08:55:38 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe1 reachability changes to unreachable.
Feb 13 08:55:38 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe1 state changes to inactive.
Feb 13 08:56:23 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe0 reachability changes to reachable.
Feb 13 08:56:23 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe0 state changes to active.
Feb 13 08:56:23 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe1 reachability changes to reachable.
Feb 13 08:56:23 ubnt ubnt-util: WLB: Load-Balance group G interface pppoe1 state changes to active.

By checking connections, I see some strange connection to 8.8.8.8, which I never set as a DNS server in anyplace.

1
sudo conntrack -L
1
2
icmp     1 13 src=**** dst=8.8.8.8 type=8 code=0 id=26922 src=8.8.8.8 dst=**** type=0 code=0 id=26922 mark=1694498816 use=1
udp      17 177 src=**** dst=8.8.8.8 sport=26528 dport=53 src=8.8.8.8 dst=**** sport=53 dport=26528 [ASSURED] mark=1686110208 use=1

After a thorough read about https://help.ubnt.com/hc/en-us/articles/205145990-EdgeMAX-Dual-WAN-Load-Balance-Feature, I get to know that the default behavior for a load balancer is to send a ping to 8.8.8.8 every minute if it fails, then set the reachability to unreachable for that network and redirect all traffic to the other network. With that knowledge, it’s an easy fix then.

1
2
set load-balance group G interface pppoe0 route-test type ping target 114.114.114.114
set load-balance group G interface pppoe1 route-test type ping target 114.114.114.114

Also, we can tune the failure count and interval parameters to make this even more robust and reliable. Then final settings for each interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
route-test {
    count {
        failure 6
        success 6
    }
    initial-delay 1
    interval 10
    type {
        ping {
            target 114.114.114.114
        }
    }
}

I hope you enjoyed reading this article, and in case you faced the same issue, and you feel this article helpful, please comment below and let me know.

Reference

Here is the full configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
firewall {
    all-ping enable
    broadcast-ping enable
    group {
        network-group PRIVATE_NETS {
            network 192.168.0.0/16
            network 172.16.0.0/12
            network 10.0.0.0/8
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians disable
    modify balance {
        rule 60 {
            action modify
            modify {
                table 11
            }
            source {
                address 192.168.1.13-192.168.1.16
            }
        }
        rule 70 {
            action modify
            modify {
                table 12
            }
            source {
                address 192.168.1.11-192.168.1.12
            }
        }
        rule 80 {
            action modify
            modify {
                lb-group G
            }
        }
    }
    options {
        mss-clamp {
            mss 1452
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        description "WAN 0"
        duplex auto
        pppoe 0 {
            default-route none
            description "China Telecommunication"
            mtu 1492
            name-server none
            password ****
            user-id ****
        }
        speed auto
    }
    ethernet eth1 {
        description "WAN 1"
        duplex auto
        pppoe 1 {
            default-route none
            description "China Unicom"
            mtu 1492
            name-server none
            password ****
            user-id ****
        }
        speed auto
    }
    ethernet eth2 {
        duplex auto
        speed auto
    }
    ethernet eth3 {
        duplex auto
        speed auto
    }
    ethernet eth4 {
        duplex auto
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        address 192.168.1.1/24
        description Local
        firewall {
            in {
                modify balance
            }
        }
        mtu 1492
        switch-port {
            interface eth2 {
            }
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
load-balance {
    group G {
        exclude-local-dns disable
        flush-on-active enable
        gateway-update-interval 20
        interface pppoe0 {
            route-test {
                count {
                    failure 6
                    success 6
                }
                initial-delay 1
                interval 10
                type {
                    ping {
                        target 114.114.114.114
                    }
                }
            }
            weight 70
        }
        interface pppoe1 {
            route-test {
                count {
                    failure 6
                    success 6
                }
                initial-delay 1
                interval 10
                type {
                    ping {
                        target 114.114.114.114
                    }
                }
            }
            weight 30
        }
        lb-local enable
        lb-local-metric-change disable
        sticky {
            source-addr enable
        }
    }
}
port-forward {
    auto-firewall disable
    hairpin-nat disable
    wan-interface pppoe0
}
protocols {
    static {
        interface-route 0.0.0.0/0 {
            next-hop-interface pppoe0 {
            }
            next-hop-interface pppoe1 {
            }
        }
        table 11 {
            interface-route 0.0.0.0/0 {
                next-hop-interface pppoe0 {
                }
            }
        }
        table 12 {
            interface-route 0.0.0.0/0 {
                next-hop-interface pppoe1 {
                }
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN {
            authoritative enable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.1.1
                lease 86400
                start 192.168.1.38 {
                    stop 192.168.1.243
                }
                static-mapping 12 {
                    ip-address 192.168.1.12
                    mac-address ****
                }
                static-mapping 11 {
                    ip-address 192.168.1.11
                    mac-address ****
                }
                static-mapping EdgeSwitch {
                    ip-address 192.168.1.2
                    mac-address ****
                }
                static-mapping 13 {
                    ip-address 192.168.1.13
                    mac-address ****
                }
                static-mapping 14 {
                    ip-address 192.168.1.14
                    mac-address ****
                }
                static-mapping 15 {
                    ip-address 192.168.1.15
                    mac-address ****
                }
                static-mapping 16 {
                    ip-address 192.168.1.16
                    mac-address ****
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 2000
            listen-on switch0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers disable
    }
    nat {
        rule 1 {
            description 11
            destination {
                port 47900-47910
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.11
                port 47900-47910
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 2 {
            description 12
            destination {
                port 47800-47810
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.12
                port 47800-47810
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 3 {
            description 13
            destination {
                port 47300-47310
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.13
                port 47300-47310
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 4 {
            description 14
            destination {
                port 47400-47410
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.14
                port 47400-47410
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 5 {
            description 15
            destination {
                port 47500-47510
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.15
                port 47500-47510
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 6 {
            description 16
            destination {
                port 47600-47610
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.16
                port 47600-47610
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 7 {
            description 11
            destination {
                port 47900-47910
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.11
                port 47900-47910
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 8 {
            description 12
            destination {
                port 47800-47810
            }
            inbound-interface pppoe0
            inside-address {
                address 192.168.1.12
                port 47800-47810
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 9 {
            description 13
            destination {
                port 47300-47310
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.13
                port 47300-47310
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 10 {
            description 14
            destination {
                port 47400-47410
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.14
                port 47400-47410
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 11 {
            description 15
            destination {
                port 47500-47510
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.15
                port 47500-47510
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 12 {
            description 16
            destination {
                port 47600-47610
            }
            inbound-interface pppoe1
            inside-address {
                address 192.168.1.16
                port 47600-47610
            }
            log disable
            protocol tcp_udp
            type destination
        }
        rule 5000 {
            description "masquerade for WAN 0"
            log disable
            outbound-interface pppoe0
            protocol all
            source {
            }
            type masquerade
        }
        rule 5002 {
            description "masquerade for WAN 1"
            log disable
            outbound-interface pppoe1
            protocol all
            source {
            }
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    ubnt-discover {
        disable
    }
    unms {
        disable
    }
}
system {
    conntrack {
        expect-table-size 4096
        hash-size 4096
        modules {
            sip {
                disable
            }
        }
        table-size 32768
        tcp {
            half-open-connections 512
            loose enable
            max-retrans 3
        }
    }
    host-name ubnt
    ipv6 {
        disable
    }
    login {
        user ubnt {
            authentication {
                encrypted-password ****
                plaintext-password ""
            }
            level admin
        }
    }
    name-server 114.114.114.114
    name-server 114.114.115.115
    name-server 119.29.29.29
    name-server 223.5.5.5
    offload {
        hwnat enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone Asia/Singapore
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:suspend@1:system@4:ubnt-pptp@1:ubnt-udapi-server@1:ubnt-unms@1:ubnt-util@1:vrrp@1:vyatta-netflow@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v2.0.8.5247496.191120.1124 */

Comments