Following guides here, I run:
CoreELEC:~ $ /usr/bin/rtk_hciattach -n -s 115200 ttyS1 rtk_h5 &
CoreELEC:~ $ Realtek Bluetooth init uart with init speed:115200, final_speed:115200, type:HCI UART H5
Realtek Bluetooth :Realtek hciattach version 2.5
Realtek Bluetooth :3-wire sync pattern resend : 1, len: 8
Realtek Bluetooth :Get SYNC Resp Pkt
Realtek Bluetooth :3-wire config pattern resend : 1 , len: 10
Realtek Bluetooth :Get SYNC pkt-active mode
Realtek Bluetooth :Get SYNC pkt-active mode
Realtek Bluetooth :Get CONFG pkt-active mode
Realtek Bluetooth :3-wire config pattern resend : 2 , len: 10
Realtek Bluetooth :Get CONFG resp pkt-active mode
Realtek Bluetooth :H5 init finished
Realtek Bluetooth :config offset(f4),length(8)
Realtek Bluetooth :config baud rate to :5002, hwflowcontrol:5f, 1
Realtek Bluetooth :config offset(27),length(1)
Realtek Bluetooth :config offset(fe),length(1)
Realtek Bluetooth :config offset(15b),length(4)
Realtek Bluetooth :config offset(1e3),length(1)
Realtek Bluetooth :config offset(db),length(1)
Realtek Bluetooth :Get config baud rate from config file:5002
Realtek Bluetooth :Load FW OK
Realtek Bluetooth :RTK send HCI_VENDOR_READ_RTK_ROM_VERISION_Command
Realtek Bluetooth :Received reliable seqno 0 from card
Realtek Bluetooth :receive hci command complete event with command:1001
Realtek Bluetooth :Read RTK LMP version with Status:0
Realtek Bluetooth :gLmpVersion = 0x8822
Realtek Bluetooth :RTK send HCI_VENDOR_READ_RTK_ROM_VERISION_Command
Realtek Bluetooth :Received reliable seqno 1 from card
Realtek Bluetooth :receive hci command complete event with command:fc6d
Realtek Bluetooth :Read RTK rom version with Status:0
Realtek Bluetooth :rtk_hw_cfg.eversion = 2
Realtek Bluetooth :rtk_get_fw_project_id: opcode 0, len 1, data 1
Realtek Bluetooth ERROR: lmp_version is 8822, project_id is 8723, does not match!!!
Realtek Bluetooth :baudrate in change speed command: 0x2 0x50 0x0 0x0
Realtek Bluetooth :Received reliable seqno 2 from card
Realtek Bluetooth :receive hci command complete event with command:fc17
Realtek Bluetooth :Change BD Rate with status:0
Realtek Bluetooth :final_speed 2000000
Realtek Bluetooth :hw flow control enable
Realtek Bluetooth :Init Process finished
Realtek Bluetooth post process
Device setup complete
CoreELEC:~ $ rfkill list
0: bt-dev: Bluetooth
Soft blocked: no
Hard blocked: no
1: phy0: Wireless LAN
Soft blocked: no
Hard blocked: no
See ERROR in the middle:
Realtek Bluetooth ERROR: lmp_version is 8822, project_id is 8723, does not match!!!
Also, here they work with GPIO to activate BT_EN pin, not sure if it needed.