Merge "Further reduce LM memory cost" into sc-dev