starrocks version:1.18.1

log: 2021-10-20 11:34:15 ERROR TThreadPoolServer:321 - Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ... 9 more

Ranger is configured for Hive but Kerberos is not enabled, how can I set User？

Steps to reproduce the behavior (Required)

1. start a cluster includes 3 FE and 4 BE, use the release here.
2. buid BE from commit 92d8297667cf0ffaeb1406c1d7443ec68e5da3df of main brach
3. replace one BE of cluster by that built from step 2
4. after about one hour, the BE got exception and exit

Expected behavior (Required)

be normal

Real behavior (Required)

I0914 17:40:26.355070 1166569 internal_service.cpp:241] exec plan fragment, fragment_instance_id=ca55666a-153f-11ec-ba58-6c92bf928229, coord=TNetworkAddress(hostname=********, port=9020), backend=7 is_pipeline 0
I0914 17:40:26.355098 1166569 plan_fragment_executor.cpp:68] Prepare(): query_id=ca55666a-153f-11ec-ba58-6c92bf928226 fragment_instance_id=ca55666a-153f-11ec-ba58-6c92bf928229 backend_num=7
I0914 17:40:26.355139 1166569 plan_fragment_executor.cpp:124] Using query memory limit: 2.00 GB
I0914 17:40:26.355481 1166418 plan_fragment_executor.cpp:228] Open(): fragment_instance_id=ca55666a-153f-11ec-ba58-6c92bf928229
src/tcmalloc.cc:332] Attempt to free invalid pointer 0x623ff80
*** Aborted at 1631612426 (unix time) try "date -d @1631612426" if you are using GNU date ***
PC: @     0x7effb7190387 __GI_raise
*** SIGABRT (@0x1f80011cb88) received by PID 1166216 (TID 0x7efc00568700) from PID 1166216; stack trace: ***
    @          0x3145132 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7effb8e3b630 (unknown)
    @     0x7effb7190387 __GI_raise
    @     0x7effb7191a78 __GI_abort
    @          0x140b0a0 _ZN8tcmalloc3LogENS_7LogModeEPKciNS_7LogItemES3_S3_S3_.cold
    @          0x4247609 (anonymous namespace)::InvalidFree()
    @          0x1edb505 starrocks::vectorized::timestamp::to_string()
    @          0x1ed9265 starrocks::vectorized::TimestampValue::to_string()
    @          0x23778c3 starrocks::cast_to_string<>()
    @          0x238a4af starrocks::ColumnValueRange<>::to_olap_filter()
    @          0x249bdc5 starrocks::vectorized::OlapScanNode::_start_scan()
    @          0x249c7ca starrocks::vectorized::OlapScanNode::get_next()
    @          0x25c0669 starrocks::vectorized::ProjectNode::get_next()
    @          0x247a6a4 starrocks::vectorized::DistinctStreamingNode::get_next()
    @          0x1ef22c7 starrocks::PlanFragmentExecutor::_get_next_internal_vectorized()
    @          0x1ef34ef starrocks::PlanFragmentExecutor::_open_internal_vectorized()
    @          0x1ef40eb starrocks::PlanFragmentExecutor::open()
    @          0x1e8f77f starrocks::FragmentExecState::execute()
    @          0x1e909a0 starrocks::FragmentMgr::exec_actual()
    @          0x1e958dc std::_Function_handler<>::_M_invoke()
    @          0x1fd8754 starrocks::ThreadPool::dispatch_thread()
    @          0x1fe6bf5 starrocks::Thread::supervise_thread()
    @     0x7effb8e33ea5 start_thread
    @     0x7effb72589fd __clone

StarRocks version (Required)

• commit 92d8297
• OS CentOS7.2
• gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)

Steps to reproduce the behavior (Required)

1. i try to connect starrocks use trino-server-355

2. this is the file content i put into trino catalog directory ，file name is doris.properties connector.name=mysql connection-url=jdbc:mysql://10.1.8.143:9030 connection-user=root connection-password= case-insensitive-name-matching=true

3. i get this with command “show catalogs” and it show my starrocks's catalag “doris” trino:dw> show catalogs; Catalog

     clickhouse 
     doris      
     hive       
     hive1      
     iceberg    
     jmx        
     kudu       
     oracle     
     system     
    (9 rows)
   

   Query 20210913_063107_00042_yjd54, FINISHED, 3 nodes Splits: 53 total, 53 done (100.00%) 0.23 [0 rows, 0B] [0 rows/s, 0B/s]

4. then i get this result with command “show shcemas from doris ;”

   trino:dw> show schemas from doris; Schema

   _statistics_       
   dw                 
   example_db         
   information_schema 
   tests              
   

   (5 rows)

   Query 20210913_063315_00043_yjd54, FINISHED, 4 nodes Splits: 53 total, 53 done (100.00%) 0.25 [5 rows, 72B] [19 rows/s, 287B/s]

5. when i want to see table ,i exec this command "show tables from doris.dw" ,but it show nothing。

   trino:dw> show tables from doris.dw;
   Table 
   

   ---

   (0 rows)

   Query 20210913_063526_00045_yjd54, FINISHED, 4 nodes Splits: 53 total, 53 done (100.00%) 0.26 [0 rows, 0B] [0 rows/s, 0B/s]

6. this is the table when i use mysql client to exec commend “show tables from dw;” MySQL [(none)]> show tables from dw; +----------------------------------+ | Tables_in_dw | +----------------------------------+ | DICTIONARY_DORIS | | GLOG_DEPOSIT | | GLOG_LOGIN | | GLOG_LOGIN_BITMAP_MID | | GLOG_LOGIN_RATIO | | GLOG_LOGOUT_AVGTIME | | GLOG_LTV | | GLOG_M_LOGIN_RT | | GLOG_M_RETAIN_RT | | GLOG_NEWLOGIN | | GLOG_NEWLOGIN_RETAIN | | GLOG_OLNUM | | ZZ_LOGIN | | ZZ_LOGIN_T | ...... +----------------------------------+ 46 rows in set (0.01 sec)

Real behavior (Required)

 i'm disturbed why trino can connect starrocks, but cant get  my tables in schema "dw" ,  also cant query table in starrocks.
 before this i update the version from 1.17 to 1.18.2  , detail step is ：
    1）replace  /fe/ bin，/fe/lib， /fe/spark-dpp  this three directory
    2）replace  /be/bin，/be/lib，/be/www  this three directory

StarRocks version (Required)

StarRocks-1.18.2

hi, I run cmd sh ./run-ut.sh --be --run in my Debian 9 linux 4.14. And got the error ./run-ut.sh: 19: set: Illegal option -o pipefail

ref: #1422

I do a experiment on ssb_100g using

• 8 scan threads and
• set parallel_fragment_exec_instance_num = 8

Before improvement

mysql> select count(s_nation) from lineorder_flat where s_nation like '%N';
+-------------------+
| count(`s_nation`) |
+-------------------+
|          72048551 |
+-------------------+
1 row in set (0.92 sec)

And using this PR, time can be reduced for 26%

mysql> select count(s_nation) from lineorder_flat where s_nation like '%N';
+-------------------+
| count(`s_nation`) |
+-------------------+
|          72048551 |
+-------------------+
1 row in set (0.68 sec)

And if we enable global dict optimization set cbo_enable_low_cardinality_optimize = true, time can be reduces further for 52%.

mysql> select count(s_nation) from lineorder_flat where s_nation like '%N';
+-------------------+
| count(`s_nation`) |
+-------------------+
|          72048551 |
+-------------------+
1 row in set (0.44 sec)

Steps to reproduce the behavior (Required)

select split("a,b,c",",");

Expected behavior (Required)

+---------------------+
| split('a,b,c', ',') |
+---------------------+
| ["a","b","c"]       |
+---------------------+

Real behavior (Required)

ERROR 1064 (HY000): Unexpected exception: org.apache.doris.catalog.ArrayType cannot be cast to org.apache.doris.catalog.ScalarType

StarRocks version (Required)

A version that works properly

1.18.2 caa8b52 
+---------------------+
| split('a,b,c', ',') |
+---------------------+
| ["a","b","c"]       |
+---------------------+

A version that does not work properly ， download from : https://www.dorisdb.com/zh-CN/download/community

1.18.2 7b65727  is Unexpected exception
ERROR 1064 (HY000): Unexpected exception: org.apache.doris.catalog.ArrayType cannot be cast to org.apache.doris.catalog.ScalarType

please check and repair it

Use another way to serialize/deserialize hahset in distinct agg state:

1. save packet size
2. better cache locality to do merge

Testing following SQL on ssb dataset, query time can be reduced from 60s -> 20s

SELECT COUNT (DISTINCT lo_orderkey), lo_linenumber FROM lineorder_flat group by lo_linenumber;

Close issue #607

With this optimization, the memory usage of 73000 tablets, each of which contains 1500 columns of type VARCHAR, dropped from 23.5GB to 7.6GB.

When I follow the documentation to install , mysql client cannot parse when I use the following command

SHOW PROC '/frontends'\\G
ERROR: Unknown command '\\'.
-> ;
ERROR 1064 (HY000): Please check your sql, we meet an error when parsing.

But this command can be executed

show backends;
+-----------+-----------------+-----------------+---------------+--------+----------+----------+---------------------+-------------- 
-------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+------- 
--+----------------+--------+----------------+--------------------------------------------------------+
| BackendId | Cluster         | IP              | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | 
Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | 
UsedPct | MaxDiskUsedPct | ErrMsg | Version        | Status                                                 |
+-----------+-----------------+-----------------+---------------+--------+----------+----------+---------------------+-------------- 
-------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+------- 
--+----------------+--------+----------------+--------------------------------------------------------+
| 10002     | default_cluster | 192.168.134.128 | 9050          | 9060   | 8040     | 8060     | 2021-09-08 15:01:37 | 2021-09-08 
15:13:47 | true  | false                | false                 | 10        | .000             | 1.364 GB      | 16.986 GB     | 91.97 % | 91.97 %        |        
| 1.18.2-caa8b52 | {"lastSuccessReportTabletsTime":"2021-09-08 15:13:37"} |

And so does this command

SHOW PROC '/frontends'\\G

Changes

Add vectorized method (send_chunk) for MemoryScratchSink to convert StarRocks in-memory column into Arrow array.

Steps to reproduce the behavior (Required)

1. DROP DATABASE IF EXISTS d32; CREATE DATABASE d32; USE d32; admin set frontend config ("enable_decimal_v3" = "true"); CREATE TABLE t0 (c_0_0 TINYINT NULL ,c_0_1 BOOLEAN NOT NULL ,c_0_2 VARCHAR(31) NULL ,c_0_3 DECIMAL64(5, 1) NULL ,c_0_4 DECIMAL128(36, 26) NOT NULL ,c_0_5 BOOLEAN NULL ,c_0_6 DATE NULL ,c_0_7 SMALLINT NULL ,c_0_8 INT NOT NULL ,c_0_9 CHAR(1) NOT NULL ,c_0_10 DATETIME NOT NULL ,c_0_11 BOOLEAN NOT NULL ,c_0_12 DECIMAL32(8, 0) NULL ,c_0_13 DECIMAL64(1, 1) NULL ,c_0_14 DECIMAL32(2, 1) NOT NULL ,c_0_15 TINYINT NULL ) UNIQUE KEY (c_0_0,c_0_1) DISTRIBUTED BY HASH (c_0_1,c_0_0) properties("replication_num"="1") ; CREATE TABLE IF NOT EXISTS t1 (c_1_0 DECIMAL64(9, 1) NOT NULL ,c_1_1 BIGINT REPLACE NULL ,c_1_2 DATE REPLACE NULL ,c_1_3 DECIMAL128(35, 21) MIN NULL ) AGGREGATE KEY (c_1_0) DISTRIBUTED BY HASH (c_1_0) properties("replication_num"="1") ; CREATE TABLE IF NOT EXISTS t2 (c_2_0 DECIMAL32(8, 2) NULL ,c_2_1 DECIMAL64(4, 2) NULL ,c_2_2 BOOLEAN NULL ,c_2_3 BOOLEAN NULL ,c_2_4 DECIMAL128(22, 14) NOT NULL ,c_2_5 FLOAT NULL ,c_2_6 DECIMAL64(5, 1) NULL ,c_2_7 DATETIME NULL ,c_2_8 INT NULL ) UNIQUE KEY (c_2_0,c_2_1,c_2_2) DISTRIBUTED BY HASH (c_2_2,c_2_0,c_2_1) properties("replication_num"="1") ; CREATE TABLE t3 (c_3_0 INT NOT NULL ,c_3_1 DATE NOT NULL ,c_3_2 DATETIME NOT NULL ,c_3_3 DECIMAL64(7, 4) NULL ,c_3_4 DECIMAL128(2, 1) NOT NULL ,c_3_5 CHAR(1) NULL ,c_3_6 BOOLEAN NULL ,c_3_7 DECIMAL128(8, 6) NOT NULL ,c_3_8 DECIMAL32(5, 0) NULL ,c_3_9 BOOLEAN NOT NULL ) UNIQUE KEY (c_3_0,c_3_1,c_3_2) DISTRIBUTED BY HASH (c_3_0,c_3_1,c_3_2) properties("replication_num"="1") ; INSERT INTO t1 VALUES (5949038.3, 63714999, "1970-01-01", 3.4254613511636E-8); INSERT INTO t2 VALUES (1539.93, 0.54, true, false, 4.7076306E-7, 0.8798927357508506, 382.2, "1970-01-01 08:00:00", 27); DELETE FROM t1 WHERE c_1_0=9309717.9; INSERT INTO t3 VALUES (7, "1970-01-01", "1970-01-01 08:00:00", 0.0160, 0.3, "", false, 0.000017, 73294, true), (8, "1970-01-01", "1970-01-01 08:00:00", 0.0637, 0.7, "", false, 0.000089, 9156, true);
2. SELECT DISTINCT subt2.c_2_0 FROM (SELECT t2.c_2_0, t2.c_2_1, t2.c_2_2, t2.c_2_3, t2.c_2_4, t2.c_2_5, t2.c_2_6, t2.c_2_7, t2.c_2_8 FROM t2 WHERE ( t2.c_2_1 ) >= ( t2.c_2_1 ) ORDER BY t2.c_2_6) subt2 LEFT SEMI JOIN t0 ON subt2.c_2_8 = t0.c_0_2 AND subt2.c_2_8 > t0.c_0_2 AND subt2.c_2_7 != t0.c_0_7 AND subt2.c_2_7 <= DAYOFMONTH("1969-12-28") , t1 RIGHT SEMI JOIN (SELECT t3.c_3_0, t3.c_3_1, t3.c_3_2, t3.c_3_3, t3.c_3_4, t3.c_3_5, t3.c_3_6, t3.c_3_7, t3.c_3_8, t3.c_3_9 FROM t3 WHERE CAST(84 AS BOOLEAN ) ) subt3 ON t1.c_1_1 = subt3.c_3_3 AND t1.c_1_1 <= subt3.c_3_3 AND t1.c_1_1 <= subt3.c_3_4 AND t1.c_1_1 <= subt3.c_3_4;

Expected behavior (Required)

query right

Real behavior (Required)

Column 'subt2.c_2_0' cannot be resolved

StarRocks version (Required)

• You can get the StarRocks version by executing SQL select current_version()
• main release 28364fc

for #1580

Feature request

SR should support mysql table sink to query result

insert into mysql_table select xxx from xxx;

lazy load zonemap/bitmap/bloomfilter index, and fix bug of reader_type, to #1554

ref: https://github.com/StarRocks/starrocks/issues/1576

create table t(id bigint not null, name string not null) primary key(id) distributed by hash(id) buckets 4 properties("bloom_filter_clumns":"name"); Failed:

Bloom filter index only used in columns of DUP_KEYS table or key columns of UNIQUE_KEYS/AGG_KEYS table

need to remove this check for primary key

Update drop backends syntax behavior. Added syntax ALTER SYSTEM DROP BACKEND <backends> FORCE.

Feature request

Is your feature request related to a problem? Please describe.

When a tablet with a single duplication table exists on a backend, deleting backend will cause table data loss.

Describe the solution you'd like

I think the drop backends operation is rarely used in online clusters. And ‘dropp’ is not a correct word. Make the following changes to the behavior of the drop backend statement:

• ALTER SYSTEM DROP <backends> can delete backend instead of DROPP.
• When there is a tablet with a single duplication table on the backend to be deleted, an exception will be thrown and ALTER SYSTEM DROP <backends> FORCE needs to be used to delete.
• The original behavior of DROPP remains unchanged, I think it could be removed in a future version.

Describe alternatives you've considered

Additional context

ref #1559

Ignore the resource which was dropped during 'show resources' to fix npe exception.

2021-11-24 15:35:16,936 WARN (starrocks-mysql-nio-pool-152|8584) [StmtExecutor.execute():454] execute Exception, sql show resources
java.lang.NullPointerException: null
        at com.starrocks.catalog.ResourceMgr$ResourceProcNode.fetchResult(ResourceMgr.java:154) ~[starrocks-fe.jar:?]
        at com.starrocks.catalog.ResourceMgr.getResourcesInfo(ResourceMgr.java:127) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ShowExecutor.handleShowResources(ShowExecutor.java:1234) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:225) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1169) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:425) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:248) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:397) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:633) ~[starrocks-fe.jar:

Steps to reproduce the behavior (Required)

1. set cbo_enable_low_cardinality_optimize = true

2. run query62.sql select substr(w_warehouse_name,1,20) ,sm_type ,web_name ,sum(case when (ws_ship_date_sk - ws_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 30) and (ws_ship_date_sk - ws_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 60) and (ws_ship_date_sk - ws_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 90) and (ws_ship_date_sk - ws_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from web_sales ,warehouse ,ship_mode ,web_site ,date_dim where d_month_seq between 1200 and 1200 + 11 and ws_ship_date_sk = d_date_sk and ws_warehouse_sk = w_warehouse_sk and ws_ship_mode_sk = sm_ship_mode_sk and ws_web_site_sk = web_site_sk group by substr(w_warehouse_name,1,20) ,sm_type ,web_name order by substr(w_warehouse_name,1,20) ,sm_type ,web_name limit 100;

3. the same query returns 100 rows when set cbo_enable_low_cardinality_optimize = false

Expected behavior (Required)

100 rows

Real behavior (Required)

ERROR 1064 (HY000): cid:4 column has a non-dictionary coding page

StarRocks version (Required)

• You can get the StarRocks version by executing SQL select current_version() branch-2.0 2ccfeaa release