Submitted by ryan.grace on April 5, 2024 - 6:36am
Problem:
Vision no data being consumed at ingest
df -k showed /tmp partition full
on investigation full with hs_err, these are java dumps due to kafka process crashing.
also in /var/log/messages we see
Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,173 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.index) has non-zero size but the last offset is 5655889921 which is no larger than the base offset5655889921.}. deleting /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.timeindex, /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.index, and /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.txnindex and rebuilding index...
Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,194 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.index) has non-zero size but the last offset is 388 which is no larger than the base offset 388.}. deleting /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.timeindex, /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.index, and /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.txnindex and rebuilding index...
Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,212 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.index) has non-zero size but the last offset is 1474807281 which is no larger than the base offset 1474807281.}. deleting /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.timeindex, /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.index, and /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.txnindex and rebuilding index...
Solution:
go to the vision cluster with the depot then run the repair on bigdata
cd /visionreleases/Vision//ansible
sh deployer.sh "bigdata" -e repair=true