failed authentication due to: Unexpected handshake request with client mechanism SCRAM-SHA-512, enabled mechanisms are [] (org.apache.kafka.clients.NetworkClient)

This is the error which was very nagging.

kafka-topics --bootstrap-server <kafka_server>:9094 --command-config /etc/
kafka/client_sasl.properties --topic k2view.dipesh.test --create
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
[2024-07-08 11:49:12,038] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (b-1.*******.ic****.c4.kafka.eu-central-1.amazonaws.com/10.128.82.xxx:9094) failed authentication due to: Unexpected handshake request with client mechanism SCRAM-SHA-512, enabled mechanisms are [] (org.apache.kafka.clients.NetworkClient)
[2024-07-08 11:49:12,040] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected handshake request with client mechanism SCRAM-SHA-512, enabled mechanisms are []
Error while executing topic command : Unexpected handshake request with client mechanism SCRAM-SHA-512, enabled mechanisms are []
[2024-07-08 11:49:12,044] ERROR org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected handshake request with client mechanism SCRAM-SHA-512, enabled mechanisms are []
 (kafka.admin.TopicCommand$)

Sometimes i also got this error while debugging:

kafka-topics — bootstrap-server <kafka_server>:9094 — topic kafkatest — create Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics [2024–07–08 12:00:26,143] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics (kafka.admin.TopicCommand$)

The solution was that i had to try the port 9096

kafka-topics --bootstrap-server b-1.******dev.******.c4.kafka.<region>.amazonaws.com:9096 --command-config /etc/kafka/client_sasl.properties --topic k2viewtest --create
Created topic k2viewtest.

And this creates the topic successfully.

The security group for the msk should look like this as shown below: