Spark: [Question]: spark.Sql insert data to hive error

Created on 31 Mar 2020  路  2Comments  路  Source: dotnet/spark

this is my codes:
private static void HiveTest() { var spark = SparkSession.Builder() .Config("spark.sql.warehouse.dir", "/user/hive/warehouse2/") .EnableHiveSupport().GetOrCreate(); //spark.Sql("create database hive_test_db"); spark.Sql("use hive_test_db"); spark.Sql("show tables").Show(); //spark.Sql("create table hive_test_db.test_table(id INT,name string)"); spark.Sql("insert into hive_test_db.test_table(id,name) values(1,'2')"); }

error:

F:\ProjectTest\HelloSpark\bin\Debug\netcoreapp3.1>spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.10.0.jar dotnet HelloSpark.dll
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/03/31 13:44:01 INFO DotnetRunner: Starting DotnetBackend with dotnet.
20/03/31 13:44:01 INFO DotnetRunner: Port number used by DotnetBackend is 64900
20/03/31 13:44:01 INFO DotnetRunner: Adding key=spark.jars and value=file:/F:/ProjectTest/HelloSpark/bin/Debug/netcoreapp3.1/microsoft-spark-2.4.x-0.10.0.jar to environment
20/03/31 13:44:01 INFO DotnetRunner: Adding key=spark.app.name and value=org.apache.spark.deploy.dotnet.DotnetRunner to environment
20/03/31 13:44:01 INFO DotnetRunner: Adding key=spark.submit.deployMode and value=client to environment
20/03/31 13:44:01 INFO DotnetRunner: Adding key=spark.master and value=local to environment
[2020-03-31T05:44:11.5393411Z] [GOSL-PC] [Info] [ConfigurationService] Using port 64900 for connection.
[2020-03-31T05:44:11.5714317Z] [GOSL-PC] [Info] [JvmBridge] JvMBridge port is 64900
20/03/31 13:44:11 INFO SparkContext: Running Spark version 2.4.5
20/03/31 13:44:11 INFO SparkContext: Submitted application: org.apache.spark.deploy.dotnet.DotnetRunner
20/03/31 13:44:11 INFO SecurityManager: Changing view acls to: lisisie
20/03/31 13:44:11 INFO SecurityManager: Changing modify acls to: lisisie
20/03/31 13:44:11 INFO SecurityManager: Changing view acls groups to:
20/03/31 13:44:11 INFO SecurityManager: Changing modify acls groups to:
20/03/31 13:44:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(lisisie); groups with view permissions: Set(); users with modify permissions: Set(lisisie); groups with modify permissions: Set()
20/03/31 13:44:11 INFO Utils: Successfully started service 'sparkDriver' on port 64915.
20/03/31 13:44:11 INFO SparkEnv: Registering MapOutputTracker
20/03/31 13:44:11 INFO SparkEnv: Registering BlockManagerMaster
20/03/31 13:44:12 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/03/31 13:44:12 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/03/31 13:44:12 INFO DiskBlockManager: Created local directory at C:\Users\lisisie\AppData\Local\Temp\blockmgr-bd03e9e4-8928-461e-9b2c-dbd150bc4a0c
20/03/31 13:44:12 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
20/03/31 13:44:12 INFO SparkEnv: Registering OutputCommitCoordinator
20/03/31 13:44:12 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
20/03/31 13:44:12 INFO Utils: Successfully started service 'SparkUI' on port 4041.
20/03/31 13:44:12 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://GOSL-PC:4041
20/03/31 13:44:12 INFO SparkContext: Added JAR file:/F:/ProjectTest/HelloSpark/bin/Debug/netcoreapp3.1/microsoft-spark-2.4.x-0.10.0.jar at spark://GOSL-PC:64915/jars/microsoft-spark-2.4.x-0.10.0.jar with timestamp 1585633452264
20/03/31 13:44:12 INFO Executor: Starting executor ID driver on host localhost
20/03/31 13:44:12 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64928.
20/03/31 13:44:12 INFO NettyBlockTransferService: Server created on GOSL-PC:64928
20/03/31 13:44:12 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/03/31 13:44:12 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, GOSL-PC, 64928, None)
20/03/31 13:44:12 INFO BlockManagerMasterEndpoint: Registering block manager GOSL-PC:64928 with 366.3 MB RAM, BlockManagerId(driver, GOSL-PC, 64928, None)
20/03/31 13:44:12 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, GOSL-PC, 64928, None)
20/03/31 13:44:12 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, GOSL-PC, 64928, None)
20/03/31 13:44:13 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/user/hive/warehouse2/').
20/03/31 13:44:13 INFO SharedState: Warehouse path is '/user/hive/warehouse2/'.
20/03/31 13:44:13 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/03/31 13:44:15 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
20/03/31 13:44:15 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
20/03/31 13:44:15 INFO ObjectStore: ObjectStore, initialize called
20/03/31 13:44:15 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
20/03/31 13:44:15 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
20/03/31 13:44:17 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
20/03/31 13:44:17 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
20/03/31 13:44:17 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
20/03/31 13:44:17 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
20/03/31 13:44:17 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
20/03/31 13:44:17 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
20/03/31 13:44:17 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
20/03/31 13:44:17 INFO ObjectStore: Initialized ObjectStore
20/03/31 13:44:18 INFO HiveMetaStore: Added admin role in metastore
20/03/31 13:44:18 INFO HiveMetaStore: Added public role in metastore
20/03/31 13:44:18 INFO HiveMetaStore: No user is added in admin role, since config is empty
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_all_databases
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_all_databases
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_functions: db=default pat=*
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_functions: db=default pat=*
20/03/31 13:44:18 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_functions: db=hive_test_db pat=*
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_functions: db=hive_test_db pat=*

20/03/31 13:44:18 INFO SessionState: Created local directory: C:/Users/XIAOXI~1/AppData/Local/Temp/9069197f-8885-444a-9f72-96254abf342a_resources
20/03/31 13:44:18 INFO SessionState: Created HDFS directory: /tmp/hive/lisisie/9069197f-8885-444a-9f72-96254abf342a
20/03/31 13:44:18 INFO SessionState: Created local directory: C:/Users/XIAOXI~1/AppData/Local/Temp/lisisie/9069197f-8885-444a-9f72-96254abf342a
20/03/31 13:44:18 INFO SessionState: Created HDFS directory: /tmp/hive/lisisie/9069197f-8885-444a-9f72-96254abf342a/_tmp_space.db
20/03/31 13:44:18 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse2/
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_database: default
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_database: default
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_database: global_temp
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_database: global_temp
20/03/31 13:44:18 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_database: hive_test_db
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_database: hive_test_db
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_database: hive_test_db
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_database: hive_test_db
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_database: hive_test_db
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_database: hive_test_db
20/03/31 13:44:18 INFO HiveMetaStore: 0: get_tables: db=hive_test_db pat=*
20/03/31 13:44:18 INFO audit: ugi=lisisie ip=unknown-ip-addr cmd=get_tables: db=hive_test_db pat=*
20/03/31 13:44:19 INFO CodeGenerator: Code generated in 158.866254 ms
20/03/31 13:44:19 INFO CodeGenerator: Code generated in 9.233136 ms
+------------+----------+-----------+
| database| tableName|isTemporary|
+------------+----------+-----------+
|hive_test_db|test_table| false|
+------------+----------+-----------+

20/03/31 13:44:19 ERROR DotnetBackendHandler: methods:
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.slf4j.Logger org.apache.spark.sql.SparkSession.log()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.DataFrameReader org.apache.spark.sql.SparkSession.read()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.stop()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.close()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.table(java.lang.String)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.table(org.apache.spark.sql.catalyst.TableIdentifier)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public java.lang.String org.apache.spark.sql.SparkSession.version()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.range(long,long,long)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.range(long,long,long,int)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.range(long,long)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.range(long)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.SparkSessionExtensions org.apache.spark.sql.SparkSession.extensions()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.streaming.StreamingQueryManager org.apache.spark.sql.SparkSession.streams()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public boolean org.apache.spark.sql.SparkSession.initializeLogIfNecessary$default$2()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.slf4j.Logger org.apache.spark.sql.SparkSession.org$apache$spark$internal$Logging$$log_()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logTrace(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logTrace(scala.Function0,java.lang.Throwable)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public java.lang.String org.apache.spark.sql.SparkSession.logName()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logInfo(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logInfo(scala.Function0,java.lang.Throwable)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logError(scala.Function0,java.lang.Throwable)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logError(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logWarning(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logWarning(scala.Function0,java.lang.Throwable)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logDebug(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.logDebug(scala.Function0,java.lang.Throwable)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public boolean org.apache.spark.sql.SparkSession.isTraceEnabled()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.RuntimeConfig org.apache.spark.sql.SparkSession.conf()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public void org.apache.spark.sql.SparkSession.initializeLogIfNecessary(boolean)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public boolean org.apache.spark.sql.SparkSession.initializeLogIfNecessary(boolean,boolean)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.catalog.Catalog org.apache.spark.sql.SparkSession.catalog()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static org.apache.spark.sql.SparkSession$Builder org.apache.spark.sql.SparkSession.builder()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.util.CallSite org.apache.spark.sql.SparkSession.org$apache$spark$sql$SparkSession$$creationSite()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public boolean org.apache.spark.sql.SparkSession.internalCreateDataFrame$default$3()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static org.apache.spark.sql.SparkSession org.apache.spark.sql.SparkSession.active()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public java.lang.Object org.apache.spark.sql.SparkSession.time(scala.Function0)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static void org.apache.spark.sql.SparkSession.clearDefaultSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public scala.collection.mutable.HashMap org.apache.spark.sql.SparkSession.initialSessionOptions()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static void org.apache.spark.sql.SparkSession.setDefaultSession(org.apache.spark.sql.SparkSession)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static void org.apache.spark.sql.SparkSession.clearActiveSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static scala.Option org.apache.spark.sql.SparkSession.getDefaultSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(org.apache.spark.rdd.RDD,java.lang.String)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(org.apache.spark.rdd.RDD,org.apache.spark.sql.types.StructType)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.baseRelationToDataFrame(org.apache.spark.sql.sources.BaseRelation)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.internalCreateDataFrame(org.apache.spark.rdd.RDD,org.apache.spark.sql.types.StructType,boolean)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.sql(java.lang.String)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.SparkSession org.apache.spark.sql.SparkSession.cloneSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.emptyDataFrame()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.SparkContext org.apache.spark.sql.SparkSession.sparkContext()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.internal.SessionState org.apache.spark.sql.SparkSession.sessionState()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static scala.Option org.apache.spark.sql.SparkSession.getActiveSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public static void org.apache.spark.sql.SparkSession.setActiveSession(org.apache.spark.sql.SparkSession)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.emptyDataset(org.apache.spark.sql.Encoder)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.ExperimentalMethods org.apache.spark.sql.SparkSession.experimental()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.rdd.RDD,org.apache.spark.sql.types.StructType)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.api.java.JavaRDD,org.apache.spark.sql.types.StructType)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(java.util.List,java.lang.Class)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.api.java.JavaRDD,java.lang.Class)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.rdd.RDD,java.lang.Class)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(java.util.List,org.apache.spark.sql.types.StructType)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(scala.collection.Seq,scala.reflect.api.TypeTags$TypeTag)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.rdd.RDD,org.apache.spark.sql.types.StructType,boolean)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataFrame(org.apache.spark.rdd.RDD,scala.reflect.api.TypeTags$TypeTag)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.SparkSession org.apache.spark.sql.SparkSession.newSession()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.SQLContext org.apache.spark.sql.SparkSession.sqlContext()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SparkSession.readStream()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.SparkSession$implicits$ org.apache.spark.sql.SparkSession.implicits()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.types.DataType org.apache.spark.sql.SparkSession.parseDataType(java.lang.String)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.util.ExecutionListenerManager org.apache.spark.sql.SparkSession.listenerManager()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.UDFRegistration org.apache.spark.sql.SparkSession.udf()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataset(java.util.List,org.apache.spark.sql.Encoder)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataset(scala.collection.Seq,org.apache.spark.sql.Encoder)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.Dataset org.apache.spark.sql.SparkSession.createDataset(org.apache.spark.rdd.RDD,org.apache.spark.sql.Encoder)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public org.apache.spark.sql.internal.SharedState org.apache.spark.sql.SparkSession.sharedState()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final void java.lang.Object.wait() throws java.lang.InterruptedException
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final void java.lang.Object.wait(long,int) throws java.lang.InterruptedException
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final native void java.lang.Object.wait(long) throws java.lang.InterruptedException
20/03/31 13:44:19 ERROR DotnetBackendHandler: public boolean java.lang.Object.equals(java.lang.Object)
20/03/31 13:44:19 ERROR DotnetBackendHandler: public java.lang.String java.lang.Object.toString()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public native int java.lang.Object.hashCode()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final native java.lang.Class java.lang.Object.getClass()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final native void java.lang.Object.notify()
20/03/31 13:44:19 ERROR DotnetBackendHandler: public final native void java.lang.Object.notifyAll()
20/03/31 13:44:19 ERROR DotnetBackendHandler: args:
20/03/31 13:44:19 ERROR DotnetBackendHandler: argType: java.lang.String, argValue: insert into hive_test_db.test_table(id,name) values(1,'2')
[2020-03-31T05:44:19.8191802Z] [GOSL-PC] [Error] [JvmBridge] JVM method execution failed: Nonstatic method sql failed for class 7 when called with 1 arguments ([Index=1, Type=String, Value=insert into hive_test_db.test_table(id,name) values(1,'2')], )
[2020-03-31T05:44:19.8192349Z] [GOSL-PC] [Error] [JvmBridge] org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'id' expecting {'(', 'SELECT', 'FROM', 'VALUES', 'TABLE', 'INSERT', 'MAP', 'REDUCE'}(line 1, pos 36)

== SQL ==
insert into hive_test_db.test_table(id,name) values(1,'2')
------------------------------------^^^

    at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:162)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.handleBackendRequest(DotnetBackendHandler.scala:102)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:29)
    at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:24)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

[2020-03-31T05:44:19.8771299Z] [GOSL-PC] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method sql failed for class 7 when called with 1 arguments ([Index=1, Type=String, Value=insert into hive_test_db.test_table(id,name) values(1,'2')], )
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
20/03/31 13:48:12 INFO ContextCleaner: Cleaned accumulator 0

question

Most helpful comment

[2020-03-31T05:44:19.8192349Z] [GOSL-PC] [Error] [JvmBridge] org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'id' expecting {'(', 'SELECT', 'FROM', 'VALUES', 'TABLE', 'INSERT', 'MAP', 'REDUCE'}(line 1, pos 36)

== SQL ==
insert into hive_test_db.test_table(id,name) values(1,'2')
------------------------------------^^^

You don't need to specify column names when you use INSERT INTO: https://docs.databricks.com/spark/latest/spark-sql/language-manual/insert.html

All 2 comments

[2020-03-31T05:44:19.8192349Z] [GOSL-PC] [Error] [JvmBridge] org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'id' expecting {'(', 'SELECT', 'FROM', 'VALUES', 'TABLE', 'INSERT', 'MAP', 'REDUCE'}(line 1, pos 36)

== SQL ==
insert into hive_test_db.test_table(id,name) values(1,'2')
------------------------------------^^^

You don't need to specify column names when you use INSERT INTO: https://docs.databricks.com/spark/latest/spark-sql/language-manual/insert.html

thanks!
it works

Was this page helpful?
0 / 5 - 0 ratings