The output from the sensor will be changing faster than the update rate for the multimeter (voltmeter) so using that will have limited value.
You can try it. With reference to this wiring diagram:
http://www.memcals.com/pdf/VRV8 Wiring Diagram.pdf
The voltmeter needs to have an impedance >10MOhm (see the spec. sheet for the meter) in order to avoid altering the signal you are trying to measure. Most will but some of the very cheapest may not.
At the PCM, inside the car, behind the LHS kick panel, find the violet wire that connects to terminal D7 (the terminal numbers are cast into the plastic connectors) and remove the 'comb' that holds the terminals in place. Then use a dressmaker's pin inserted into the back of the plastic connector to provide contact with the D7 terminal.
Put the positive/red test terminal onto that and the negative/black terminal onto an earth like the car body.
Make sure that the test terminal banana plugs are in the correct holes to measure DC voltage. The test range setting should be 2V DC for most meters.
Run the engine.
When cold, the sensor should put out a steady 0.45V = 450mV. As it warms up, you should see the voltage start to change through about 150mV to 850mV.
As a rough idea of whether or not it is responding appropriately, see what happens to the measured voltage when you disconnect (and plug) the vacuum hose to the fuel pressure regulator - the voltage should rise = rich and disconnect the brake booster hose - the voltage should fall = lean.
You can also check the signal in the same way directly at the O2 sensor instead of at pin D7. If the voltages differ, the problem is in the wiring between the two.
If you get a voltage reading from the sensor while earthing on the car body, the earth for the sensor may be faulty. Try backprobing across the terminals at the sensor rather than using an external earth.
A better (inexpensive) indication of what the O2 sensor is doing is to buy and make one of these:
Jaycar Electronics
That will give an indication of how quickly the O2 sensor is switching from rich to lean, is fairly easy to make even as a first project but will probably need some set up to centre the oscillations.