Is giving out false information that you believe to be true a lie? Also I don't think a language model can really 'lie' as it doesn't even know what it's saying.
You're getting metaphysical. It doesn't matter if you believe to be true, if you convey false information it is a lie. But don't take it from me. Here's a dictionary definition.
~~~
lie (lies, present participle lying; simple past and past participle lied)
(intransitive) To convey a false image or impression.
Photographs often lie.
> So did the holocaust happen?
*insert muh six gorillion*
> but you just previously admitted you’re a liar.
Lmao 5D chess